课后练习 之 第五课——Week1

目录

0 环境配置

1 前言

2 一步步构建你的递归神经网络

2.0 导包

2.1 基本RNN的正向传播

RNN Cell

RNN前向传播

2.2 LSTM的前向传播

LSTM Cell

LSTM前向传播

2.3 RNN的反向传播

RNN Cell反向传播

基本RNN反向传播 

LSTM Cell反向传播

LSTM反向传播

3 字符级语言模型-恐龙岛

3.0 导包

3.1 问题重述

数据集与预处理

模型概述

3.2 构建模型的小块

裁剪优化循环中的梯度

采样

3.3 构建语言模型

梯度下降法

模型训练

3.4 结论

3.5 像莎士比亚一样写作

4 用LSTM网络即兴演奏爵士独奏

4.0 导包

4.1 问题重述

数据集

模型概述

4.2 构造模型

创建模型对象

编译模型

初始化隐藏状态和单元状态

训练模型

预测输出

4.3 生成音乐

预测与采样

生成音乐

0 环境配置

如果你在开始本节前遇到了环境配置方面的一些问题，欢迎阅读我之前写的一篇很详细文章：

⭐⭐[ pytorch+tensorflow ]⭐⭐配置两大框架下的GPU训练环境

1 前言

本文为2021吴恩达学习笔记deeplearning.ai《深度学习专项课程》篇——“第五课——Week1”章节的课后练习，完整内容参见：

深度学习入门指南——2021吴恩达学习笔记deeplearning.ai《深度学习专项课程》篇

2 一步步构建你的递归神经网络

欢迎来到课程5的第一份作业，在这里你将在NumPy中实现递归神经网络（RNN）的关键组件！

2.0 导包

  1. import numpy as np
  2. from rnn_utils import *
  3. from public_tests import *

2.1 基本RNN的正向传播

下面是实现RNN的方法，步骤:

1. 实现RNN一个时间步长所需的计算。

2. 在$T_x$时间步上实现一个循环，以便处理所有输入，一次一个。

RNN Cell

你可以把循环神经网络看作是对单个细胞的重复使用。首先，您将实现单个时间步长的计算。下图描述了RNN单元的单个时间步的操作：

  1. # UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: rnn_cell_forward
  3.  
  4. def rnn_cell_forward(xt, a_prev, parameters):
  5.     
  6.     # Retrieve parameters from "parameters"
  7.     Wax = parameters["Wax"]
  8.     Waa = parameters["Waa"]
  9.     Wya = parameters["Wya"]
  10.     ba = parameters["ba"]
  11.     by = parameters["by"]
  12.     
  13.     # compute next activation state using the formula given above
  14.     a_next = np.tanh(np.dot(Waa,a_prev) + np.dot(Wax,xt) + ba)
  15.     # compute output of the current cell using the formula given above
  16.     yt_pred = softmax( np.dot(Wya,a_next) + by)
  17.     
  18.     # store values you need for backward propagation in cache
  19.     cache = (a_next, a_prev, xt, parameters)
  20.     
  21.     return a_next, yt_pred, cache
  1. np.random.seed(1)
  2. xt_tmp = np.random.randn(3, 10)
  3. a_prev_tmp = np.random.randn(5, 10)
  4. parameters_tmp = {}
  5. parameters_tmp['Waa'] = np.random.randn(5, 5)
  6. parameters_tmp['Wax'] = np.random.randn(5, 3)
  7. parameters_tmp['Wya'] = np.random.randn(2, 5)
  8. parameters_tmp['ba'] = np.random.randn(5, 1)
  9. parameters_tmp['by'] = np.random.randn(2, 1)
  10.  
  11. a_next_tmp, yt_pred_tmp, cache_tmp = rnn_cell_forward(xt_tmp, a_prev_tmp, parameters_tmp)
  12. print("a_next[4] = \n", a_next_tmp[4])
  13. print("a_next.shape = \n", a_next_tmp.shape)
  14. print("yt_pred[1] =\n", yt_pred_tmp[1])
  15. print("yt_pred.shape = \n", yt_pred_tmp.shape)
  16.  
  17. # UNIT TESTS
  18. rnn_cell_forward_tests(rnn_cell_forward)

a_next[4] = 
 [ 0.59584544  0.18141802  0.61311866  0.99808218  0.85016201  0.99980978
 -0.18887155  0.99815551  0.6531151   0.82872037]
a_next.shape = 
 (5, 10)
yt_pred[1] =
 [0.9888161  0.01682021 0.21140899 0.36817467 0.98988387 0.88945212
 0.36920224 0.9966312  0.9982559  0.17746526]
yt_pred.shape = 
 (2, 10)
All tests passed

RNN前向传播

  1. # UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: rnn_forward
  3.  
  4. def rnn_forward(x, a0, parameters):
  5.     
  6.     # Initialize "caches" which will contain the list of all caches
  7.     caches = []
  8.     
  9.     # Retrieve dimensions from shapes of x and parameters["Wya"]
  10.     n_x, m, T_x = x.shape
  11.     n_y, n_a = parameters["Wya"].shape
  12.     
  13.     # initialize "a" and "y_pred" with zeros (≈2 lines)
  14.     a = np.zeros((n_a, m, T_x))
  15.     y_pred = np.zeros((n_y, m, T_x))
  16.     
  17.     # Initialize a_next (≈1 line)
  18.     a_next = a0
  19.     
  20.     # loop over all time-steps
  21.     for t in range(T_x):
  22.         # Update next hidden state, compute the prediction, get the cache (≈1 line)
  23.         a_next, yt_pred, cache = rnn_cell_forward(x[:,:,t], a_next, parameters)
  24.         # Save the value of the new "next" hidden state in a (≈1 line)
  25.         a[:,:,t] = a_next
  26.         # Save the value of the prediction in y (≈1 line)
  27.         y_pred[:,:,t] = yt_pred
  28.         # Append "cache" to "caches" (≈1 line)
  29.         caches.append(cache)
  30.     
  31.     # store values needed for backward propagation in cache
  32.     caches = (caches, x)
  33.     
  34.     return a, y_pred, caches
  1. np.random.seed(1)
  2. x_tmp = np.random.randn(3, 10, 4)
  3. a0_tmp = np.random.randn(5, 10)
  4. parameters_tmp = {}
  5. parameters_tmp['Waa'] = np.random.randn(5, 5)
  6. parameters_tmp['Wax'] = np.random.randn(5, 3)
  7. parameters_tmp['Wya'] = np.random.randn(2, 5)
  8. parameters_tmp['ba'] = np.random.randn(5, 1)
  9. parameters_tmp['by'] = np.random.randn(2, 1)
  10.  
  11. a_tmp, y_pred_tmp, caches_tmp = rnn_forward(x_tmp, a0_tmp, parameters_tmp)
  12. print("a[4][1] = \n", a_tmp[4][1])
  13. print("a.shape = \n", a_tmp.shape)
  14. print("y_pred[1][3] =\n", y_pred_tmp[1][3])
  15. print("y_pred.shape = \n", y_pred_tmp.shape)
  16. print("caches[1][1][3] =\n", caches_tmp[1][1][3])
  17. print("len(caches) = \n", len(caches_tmp))
  18.  
  19. #UNIT TEST    
  20. rnn_forward_test(rnn_forward)

a[4][1] = 
 [-0.99999375  0.77911235 -0.99861469 -0.99833267]
a.shape = 
 (5, 10, 4)
y_pred[1][3] =
 [0.79560373 0.86224861 0.11118257 0.81515947]
y_pred.shape = 
 (2, 10, 4)
caches[1][1][3] =
 [-1.1425182  -0.34934272 -0.20889423  0.58662319]
len(caches) = 
 2
All tests passed

恭喜! 您已经成功地从头开始构建了递归神经网络的前向传播。不错的工作!

这种RNN表现更好的情况：

  1. - 这对某些应用程序来说足够好了，但它会受到渐变消失的影响。
  2. - 当每个输出$\hat{y}^{\langle t \rangle}$可以使用“本地”上下文估计时，RNN工作得最好。
  3. - “本地”上下文指的是接近预测时间步长$t$的信息。
  4. - 更正式地说，本地上下文指的是输入$x^{\langle t' \rangle}$和预测$\hat{y}^{\langle t \rangle}$，其中$t'$接近$t$。

在下一节中，您将构建一个更复杂的模型LSTM，它更擅长处理逐渐消失的梯度。LSTM能够更好地记住一条信息并将其保存许多时间步。

2.2 LSTM的前向传播

LSTM单元的操作如下图所示：

与上面的RNN示例类似，您将首先为单个时间步实现LSTM单元。然后，您将在“for循环”中迭代地调用它，以使它处理具有$T_x$时间步长的输入。 

LSTM Cell
  1. # UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: lstm_cell_forward
  3.  
  4. def lstm_cell_forward(xt, a_prev, c_prev, parameters):
  5.  
  6.     # Retrieve parameters from "parameters"
  7.     Wf = parameters["Wf"] # forget gate weight
  8.     bf = parameters["bf"]
  9.     Wi = parameters["Wi"] # update gate weight (notice the variable name)
  10.     bi = parameters["bi"] # (notice the variable name)
  11.     Wc = parameters["Wc"] # candidate value weight
  12.     bc = parameters["bc"]
  13.     Wo = parameters["Wo"] # output gate weight
  14.     bo = parameters["bo"]
  15.     Wy = parameters["Wy"] # prediction weight
  16.     by = parameters["by"]
  17.     
  18.     # Retrieve dimensions from shapes of xt and Wy
  19.     n_x, m = xt.shape
  20.     n_y, n_a = Wy.shape
  21.  
  22.     # Concatenate a_prev and xt (≈1 line)
  23.     concat = np.concatenate([a_prev,xt])
  24.  
  25.     # Compute values for ft, it, cct, c_next, ot, a_next using the formulas given figure (4) (≈6 lines)
  26.     ft = sigmoid(np.dot(Wf,concat) + bf) # Forget Gate
  27.     it = sigmoid(np.dot(Wi,concat) + bi) # Update Gate
  28.     cct = np.tanh(np.dot(Wc,concat) + bc) # Candidate Value
  29.     c_next = c_prev*ft + cct*it # C_t
  30.     ot = sigmoid(np.dot(Wo,concat) + bo) # output gate
  31.     a_next = ot*(np.tanh(c_next)) #a_t
  32.     
  33.     # Compute prediction of the LSTM cell (≈1 line)
  34.     yt_pred = softmax(np.dot(Wy,a_next) + by)
  35.  
  36.     # store values needed for backward propagation in cache
  37.     cache = (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters)
  38.  
  39.     return a_next, c_next, yt_pred, cache
  1. np.random.seed(1)
  2. xt_tmp = np.random.randn(3, 10)
  3. a_prev_tmp = np.random.randn(5, 10)
  4. c_prev_tmp = np.random.randn(5, 10)
  5. parameters_tmp = {}
  6. parameters_tmp['Wf'] = np.random.randn(5, 5 + 3)
  7. parameters_tmp['bf'] = np.random.randn(5, 1)
  8. parameters_tmp['Wi'] = np.random.randn(5, 5 + 3)
  9. parameters_tmp['bi'] = np.random.randn(5, 1)
  10. parameters_tmp['Wo'] = np.random.randn(5, 5 + 3)
  11. parameters_tmp['bo'] = np.random.randn(5, 1)
  12. parameters_tmp['Wc'] = np.random.randn(5, 5 + 3)
  13. parameters_tmp['bc'] = np.random.randn(5, 1)
  14. parameters_tmp['Wy'] = np.random.randn(2, 5)
  15. parameters_tmp['by'] = np.random.randn(2, 1)
  16.  
  17. a_next_tmp, c_next_tmp, yt_tmp, cache_tmp = lstm_cell_forward(xt_tmp, a_prev_tmp, c_prev_tmp, parameters_tmp)
  18.  
  19. print("a_next[4] = \n", a_next_tmp[4])
  20. print("a_next.shape = ", a_next_tmp.shape)
  21. print("c_next[2] = \n", c_next_tmp[2])
  22. print("c_next.shape = ", c_next_tmp.shape)
  23. print("yt[1] =", yt_tmp[1])
  24. print("yt.shape = ", yt_tmp.shape)
  25. print("cache[1][3] =\n", cache_tmp[1][3])
  26. print("len(cache) = ", len(cache_tmp))
  27.  
  28. # UNIT TEST
  29. lstm_cell_forward_test(lstm_cell_forward)

a_next[4] = 
 [-0.66408471  0.0036921   0.02088357  0.22834167 -0.85575339  0.00138482
  0.76566531  0.34631421 -0.00215674  0.43827275]
a_next.shape =  (5, 10)
c_next[2] = 
 [ 0.63267805  1.00570849  0.35504474  0.20690913 -1.64566718  0.11832942
  0.76449811 -0.0981561  -0.74348425 -0.26810932]
c_next.shape =  (5, 10)
yt[1] = [0.79913913 0.15986619 0.22412122 0.15606108 0.97057211 0.31146381
 0.00943007 0.12666353 0.39380172 0.07828381]
yt.shape =  (2, 10)
cache[1][3] =
 [-0.16263996  1.03729328  0.72938082 -0.54101719  0.02752074 -0.30821874
  0.07651101 -1.03752894  1.41219977 -0.37647422]
len(cache) =  10
All tests passed

LSTM前向传播

现在您已经实现了LSTM的一个步骤，您可以使用for循环对它进行迭代，以处理$T_x$输入序列。

  1. # UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: lstm_forward
  3.  
  4. def lstm_forward(x, a0, parameters):
  5.  
  6.     # Initialize "caches", which will track the list of all the caches
  7.     caches = []
  8.     
  9.     #Wy = parameters['Wy'] # Save parameters in local variables in case you want to use Wy instead of parameters['Wy']
  10.     # Retrieve dimensions from shapes of x and parameters['Wy'] (≈2 lines)
  11.     n_x, m, T_x = x.shape
  12.     n_y, n_a = parameters['Wy'].shape
  13.     
  14.     # initialize "a", "c" and "y" with zeros (≈3 lines)
  15.     a = np.zeros((n_a,m,T_x))
  16.     c = np.zeros((n_a,m,T_x))
  17.     y = np.zeros((n_y,m,T_x))
  18.     
  19.     # Initialize a_next and c_next (≈2 lines)
  20.     a_next = a0
  21.     c_next = np.zeros((n_a,m))
  22.     
  23.     # loop over all time-steps
  24.     for t in range(T_x):
  25.         # Get the 2D slice 'xt' from the 3D input 'x' at time step 't'
  26.         xt = x[:,:,t]
  27.         # Update next hidden state, next memory state, compute the prediction, get the cache (≈1 line)
  28.         a_next, c_next, yt, cache = lstm_cell_forward(xt, a_next, c_next, parameters)
  29.         # Save the value of the new "next" hidden state in a (≈1 line)
  30.         a[:,:,t] = a_next
  31.         # Save the value of the next cell state (≈1 line)
  32.         c[:,:,t]  = c_next
  33.         # Save the value of the prediction in y (≈1 line)
  34.         y[:,:,t] = yt
  35.         # Append the cache into caches (≈1 line)
  36.         caches.append(cache)
  37.     
  38.     # store values needed for backward propagation in cache
  39.     caches = (caches, x)
  40.  
  41.     return a, y, c, caches
  1. np.random.seed(1)
  2. x_tmp = np.random.randn(3, 10, 7)
  3. a0_tmp = np.random.randn(5, 10)
  4. parameters_tmp = {}
  5. parameters_tmp['Wf'] = np.random.randn(5, 5 + 3)
  6. parameters_tmp['bf'] = np.random.randn(5, 1)
  7. parameters_tmp['Wi'] = np.random.randn(5, 5 + 3)
  8. parameters_tmp['bi']= np.random.randn(5, 1)
  9. parameters_tmp['Wo'] = np.random.randn(5, 5 + 3)
  10. parameters_tmp['bo'] = np.random.randn(5, 1)
  11. parameters_tmp['Wc'] = np.random.randn(5, 5 + 3)
  12. parameters_tmp['bc'] = np.random.randn(5, 1)
  13. parameters_tmp['Wy'] = np.random.randn(2, 5)
  14. parameters_tmp['by'] = np.random.randn(2, 1)
  15.  
  16. a_tmp, y_tmp, c_tmp, caches_tmp = lstm_forward(x_tmp, a0_tmp, parameters_tmp)
  17. print("a[4][3][6] = ", a_tmp[4][3][6])
  18. print("a.shape = ", a_tmp.shape)
  19. print("y[1][4][3] =", y_tmp[1][4][3])
  20. print("y.shape = ", y_tmp.shape)
  21. print("caches[1][1][1] =\n", caches_tmp[1][1][1])
  22. print("c[1][2][1]", c_tmp[1][2][1])
  23. print("len(caches) = ", len(caches_tmp))
  24.  
  25. # UNIT TEST    
  26. lstm_forward_test(lstm_forward)

a[4][3][6] =  0.17211776753291672
a.shape =  (5, 10, 7)
y[1][4][3] = 0.9508734618501101
y.shape =  (2, 10, 7)
caches[1][1][1] =
 [ 0.82797464  0.23009474  0.76201118 -0.22232814 -0.20075807  0.18656139
  0.41005165]
c[1][2][1] -0.8555449167181981
len(caches) =  2
All tests passed

恭喜你!

现在，您已经为基本RNN和LSTM实现了正向传递。当使用深度学习框架时，实现前向传递足以构建实现出色性能的系统。框架会处理好剩下的事情。

2.3 RNN的反向传播

在现代深度学习框架中，你只需要实现前向传递，框架负责后向传递，所以大多数深度学习工程师不需要为后向传递的细节而烦恼。但是，如果您是微积分专家（或者只是好奇），并且希望了解rnn中的反向反向的详细信息，您可以完成接下来的内容

RNN Cell反向传播

首先计算基本RNN单元的反向传递。然后，在下面的部分中，遍历单元格。

  1. # UNGRADED FUNCTION: rnn_cell_backward
  2.  
  3. def rnn_cell_backward(da_next, cache):
  4.  
  5.     # Retrieve values from cache
  6.     (a_next, a_prev, xt, parameters) = cache
  7.  
  8.     # Retrieve values from parameters
  9.     Wax = parameters["Wax"]
  10.     Waa = parameters["Waa"]
  11.     Wya = parameters["Wya"]
  12.     ba = parameters["ba"]
  13.     by = parameters["by"]
  14.  
  15.     # compute the gradient of tanh with respect to a_next (≈1 line)
  16.     dtanh = da_next * (
  17.         1 - np.square(np.tanh(np.dot(Wax, xt) + np.dot(Waa, a_prev) + ba))
  18.     )
  19.  
  20.     # compute the gradient of the loss with respect to Wax (≈2 lines)
  21.     dxt = np.dot(Wax.T, dtanh)
  22.     dWax = np.dot(dtanh, xt.T)
  23.  
  24.     # compute the gradient with respect to Waa (≈2 lines)
  25.     da_prev = np.dot(Waa.T, dtanh)
  26.     dWaa = np.dot(dtanh, a_prev.T)
  27.  
  28.     # compute the gradient with respect to b (≈1 line)
  29.     dba = np.sum(dtanh, axis=1, keepdims=True)
  30.  
  31.     # Store the gradients in a python dictionary
  32.     gradients = {"dxt": dxt, "da_prev": da_prev, "dWax": dWax, "dWaa": dWaa, "dba": dba}
  33.  
  34.     return gradients
  1. np.random.seed(1)
  2. xt_tmp = np.random.randn(3,10)
  3. a_prev_tmp = np.random.randn(5,10)
  4. parameters_tmp = {}
  5. parameters_tmp['Wax'] = np.random.randn(5,3)
  6. parameters_tmp['Waa'] = np.random.randn(5,5)
  7. parameters_tmp['Wya'] = np.random.randn(2,5)
  8. parameters_tmp['ba'] = np.random.randn(5,1)
  9. parameters_tmp['by'] = np.random.randn(2,1)
  10.  
  11. a_next_tmp, yt_tmp, cache_tmp = rnn_cell_forward(xt_tmp, a_prev_tmp, parameters_tmp)
  12.  
  13. da_next_tmp = np.random.randn(5,10)
  14. gradients_tmp = rnn_cell_backward(da_next_tmp, cache_tmp)
  15. print("gradients[\"dxt\"][1][2] =", gradients_tmp["dxt"][1][2])
  16. print("gradients[\"dxt\"].shape =", gradients_tmp["dxt"].shape)
  17. print("gradients[\"da_prev\"][2][3] =", gradients_tmp["da_prev"][2][3])
  18. print("gradients[\"da_prev\"].shape =", gradients_tmp["da_prev"].shape)
  19. print("gradients[\"dWax\"][3][1] =", gradients_tmp["dWax"][3][1])
  20. print("gradients[\"dWax\"].shape =", gradients_tmp["dWax"].shape)
  21. print("gradients[\"dWaa\"][1][2] =", gradients_tmp["dWaa"][1][2])
  22. print("gradients[\"dWaa\"].shape =", gradients_tmp["dWaa"].shape)
  23. print("gradients[\"dba\"][4] =", gradients_tmp["dba"][4])
  24. print("gradients[\"dba\"].shape =", gradients_tmp["dba"].shape)

gradients["dxt"][1][2] = -1.3872130506020928
gradients["dxt"].shape = (3, 10)
gradients["da_prev"][2][3] = -0.15239949377395473
gradients["da_prev"].shape = (5, 10)
gradients["dWax"][3][1] = 0.41077282493545836
gradients["dWax"].shape = (5, 3)
gradients["dWaa"][1][2] = 1.1503450668497135
gradients["dWaa"].shape = (5, 5)
gradients["dba"][4] = [0.20023491]
gradients["dba"].shape = (5, 1)

基本RNN反向传播 
  1. # UNGRADED FUNCTION: rnn_backward
  2.  
  3. def rnn_backward(da, caches):
  4.  
  5.     # Retrieve values from the first cache (t=1) of caches (≈2 lines)
  6.     (caches, x) = caches
  7.     (a1, a0, x1, parameters) = caches[0]
  8.  
  9.     # Retrieve dimensions from da's and x1's shapes (≈2 lines)
  10.     n_a, m, T_x = da.shape
  11.     n_x, m = x1.shape 
  12.  
  13.     # initialize the gradients with the right sizes (≈6 lines)
  14.     dx = np.zeros((n_x, m, T_x))
  15.     dWax = np.zeros((n_a, n_x))
  16.     dWaa = np.zeros((n_a, n_a))
  17.     dba = np.zeros((n_a, 1))
  18.     da0 = np.zeros((n_a, m))
  19.     da_prevt = np.zeros((n_a, 1))
  20.  
  21.     # Loop through all the time steps
  22.     for t in reversed(range(T_x)):
  23.         # Compute gradients at time step t. Choose wisely the "da_next" and the "cache" to use in the backward propagation step. (≈1 line)
  24.         gradients = rnn_cell_backward(da[:, :, t] + da_prevt, caches[t])
  25.         # Retrieve derivatives from gradients (≈ 1 line)
  26.         dxt, da_prevt, dWaxt, dWaat, dbat = (
  27.             gradients["dxt"],
  28.             gradients["da_prev"],
  29.             gradients["dWax"],
  30.             gradients["dWaa"],
  31.             gradients["dba"],
  32.         )
  33.         # Increment global derivatives w.r.t parameters by adding their derivative at time-step t (≈4 lines)
  34.         dx[:, :, t] = dxt
  35.         dWax += dWaxt
  36.         dWaa += dWaat
  37.         dba += dbat
  38.  
  39.     # Set da0 to the gradient of a which has been backpropagated through all time-steps (≈1 line)
  40.     da0 = da_prevt
  41.  
  42.     # Store the gradients in a python dictionary
  43.     gradients = {"dx": dx, "da0": da0, "dWax": dWax, "dWaa": dWaa,"dba": dba}
  44.  
  45.     return gradients
  1. np.random.seed(1)
  2. x_tmp = np.random.randn(3,10,4)
  3. a0_tmp = np.random.randn(5,10)
  4. parameters_tmp = {}
  5. parameters_tmp['Wax'] = np.random.randn(5,3)
  6. parameters_tmp['Waa'] = np.random.randn(5,5)
  7. parameters_tmp['Wya'] = np.random.randn(2,5)
  8. parameters_tmp['ba'] = np.random.randn(5,1)
  9. parameters_tmp['by'] = np.random.randn(2,1)
  10.  
  11. a_tmp, y_tmp, caches_tmp = rnn_forward(x_tmp, a0_tmp, parameters_tmp)
  12. da_tmp = np.random.randn(5, 10, 4)
  13. gradients_tmp = rnn_backward(da_tmp, caches_tmp)
  14.  
  15. print("gradients[\"dx\"][1][2] =", gradients_tmp["dx"][1][2])
  16. print("gradients[\"dx\"].shape =", gradients_tmp["dx"].shape)
  17. print("gradients[\"da0\"][2][3] =", gradients_tmp["da0"][2][3])
  18. print("gradients[\"da0\"].shape =", gradients_tmp["da0"].shape)
  19. print("gradients[\"dWax\"][3][1] =", gradients_tmp["dWax"][3][1])
  20. print("gradients[\"dWax\"].shape =", gradients_tmp["dWax"].shape)
  21. print("gradients[\"dWaa\"][1][2] =", gradients_tmp["dWaa"][1][2])
  22. print("gradients[\"dWaa\"].shape =", gradients_tmp["dWaa"].shape)
  23. print("gradients[\"dba\"][4] =", gradients_tmp["dba"][4])
  24. print("gradients[\"dba\"].shape =", gradients_tmp["dba"].shape)

gradients["dx"][1][2] = [-2.07101689 -0.59255627  0.02466855  0.01483317]
gradients["dx"].shape = (3, 10, 4)
gradients["da0"][2][3] = -0.31494237512664996
gradients["da0"].shape = (5, 10)
gradients["dWax"][3][1] = 11.264104496527777
gradients["dWax"].shape = (5, 3)
gradients["dWaa"][1][2] = 2.3033331265798935
gradients["dWaa"].shape = (5, 5)
gradients["dba"][4] = [-0.74747722]
gradients["dba"].shape = (5, 1) 

LSTM Cell反向传播

LSTM向后传递比向前传递稍微复杂一些。

  1. # UNGRADED FUNCTION: lstm_cell_backward
  2.  
  3. def lstm_cell_backward(da_next, dc_next, cache):
  4.  
  5.     # Retrieve information from "cache"
  6.     (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters) = cache
  7.     
  8.     # Retrieve information from "cache"
  9.     (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters) = cache
  10.     
  11.     ### START CODE HERE ###
  12.     # Retrieve dimensions from xt's and a_next's shape (≈2 lines)
  13.     n_x, m = xt.shape
  14.     n_a, m = a_next.shape
  15.     
  16.     # Compute gates related derivatives, you can find their values can be found by looking carefully at equations (7) to (10) (≈4 lines)
  17.     dot = da_next * np.tanh(c_next) * ot * (1-ot)
  18.     dcct = (dc_next * it + ot * (1 - np.square(np.tanh(c_next))) * it * da_next) * (1 - np.square(cct))
  19.     dit = (dc_next * cct + ot * (1 - np.square(np.tanh(c_next))) * cct * da_next) * it * (1 - it)
  20.     dft = (dc_next * c_prev + ot * (1 - np.square(np.tanh(c_next))) * c_prev * da_next) * ft * (1 - ft)
  21.  
  22.     # Compute parameters related derivatives. Use equations (11)-(14) (≈8 lines)
  23.     concat = np.concatenate((a_prev, xt), axis=0).T
  24.     dWf = np.dot(dft, concat)
  25.     dWi = np.dot(dit, concat)
  26.     dWc = np.dot(dcct, concat)
  27.     dWo = np.dot(dot, concat)
  28.     dbf = np.sum(dft, axis=1, keepdims=True)
  29.     dbi = np.sum(dit, axis=1, keepdims=True)
  30.     dbc = np.sum(dcct, axis=1, keepdims=True)
  31.     dbo = np.sum(dot, axis=1, keepdims=True)
  32.  
  33.     # Compute derivatives w.r.t previous hidden state, previous memory state and input. Use equations (15)-(17). (≈3 lines)
  34.     da_prev = np.dot(parameters['Wf'][:,:n_a].T, dft) +  np.dot(parameters["Wi"][:, :n_a].T, dit) + np.dot(parameters['Wc'][:,:n_a].T, dcct) + np.dot(parameters['Wo'][:,:n_a].T, dot)
  35.     dc_prev = dc_next * ft + ot * (1-np.square(np.tanh(c_next))) * ft * da_next
  36.     dxt = np.dot(parameters['Wf'][:, n_a:].T, dft) + np.dot(parameters["Wi"][:, n_a:].T, dit)+ np.dot(parameters['Wc'][:,n_a:].T,dcct) + np.dot(parameters['Wo'][:,n_a:].T, dot)
  37.     
  38.     # Save gradients in dictionary
  39.     gradients = {"dxt": dxt, "da_prev": da_prev, "dc_prev": dc_prev, "dWf": dWf,"dbf": dbf, "dWi": dWi,"dbi": dbi,
  40.                 "dWc": dWc,"dbc": dbc, "dWo": dWo,"dbo": dbo}
  41.  
  42.     return gradients
  1. np.random.seed(1)
  2. xt_tmp = np.random.randn(3,10)
  3. a_prev_tmp = np.random.randn(5,10)
  4. c_prev_tmp = np.random.randn(5,10)
  5. parameters_tmp = {}
  6. parameters_tmp['Wf'] = np.random.randn(5, 5+3)
  7. parameters_tmp['bf'] = np.random.randn(5,1)
  8. parameters_tmp['Wi'] = np.random.randn(5, 5+3)
  9. parameters_tmp['bi'] = np.random.randn(5,1)
  10. parameters_tmp['Wo'] = np.random.randn(5, 5+3)
  11. parameters_tmp['bo'] = np.random.randn(5,1)
  12. parameters_tmp['Wc'] = np.random.randn(5, 5+3)
  13. parameters_tmp['bc'] = np.random.randn(5,1)
  14. parameters_tmp['Wy'] = np.random.randn(2,5)
  15. parameters_tmp['by'] = np.random.randn(2,1)
  16.  
  17. a_next_tmp, c_next_tmp, yt_tmp, cache_tmp = lstm_cell_forward(xt_tmp, a_prev_tmp, c_prev_tmp, parameters_tmp)
  18.  
  19. da_next_tmp = np.random.randn(5,10)
  20. dc_next_tmp = np.random.randn(5,10)
  21. gradients_tmp = lstm_cell_backward(da_next_tmp, dc_next_tmp, cache_tmp)
  22. print("gradients[\"dxt\"][1][2] =", gradients_tmp["dxt"][1][2])
  23. print("gradients[\"dxt\"].shape =", gradients_tmp["dxt"].shape)
  24. print("gradients[\"da_prev\"][2][3] =", gradients_tmp["da_prev"][2][3])
  25. print("gradients[\"da_prev\"].shape =", gradients_tmp["da_prev"].shape)
  26. print("gradients[\"dc_prev\"][2][3] =", gradients_tmp["dc_prev"][2][3])
  27. print("gradients[\"dc_prev\"].shape =", gradients_tmp["dc_prev"].shape)
  28. print("gradients[\"dWf\"][3][1] =", gradients_tmp["dWf"][3][1])
  29. print("gradients[\"dWf\"].shape =", gradients_tmp["dWf"].shape)
  30. print("gradients[\"dWi\"][1][2] =", gradients_tmp["dWi"][1][2])
  31. print("gradients[\"dWi\"].shape =", gradients_tmp["dWi"].shape)
  32. print("gradients[\"dWc\"][3][1] =", gradients_tmp["dWc"][3][1])
  33. print("gradients[\"dWc\"].shape =", gradients_tmp["dWc"].shape)
  34. print("gradients[\"dWo\"][1][2] =", gradients_tmp["dWo"][1][2])
  35. print("gradients[\"dWo\"].shape =", gradients_tmp["dWo"].shape)
  36. print("gradients[\"dbf\"][4] =", gradients_tmp["dbf"][4])
  37. print("gradients[\"dbf\"].shape =", gradients_tmp["dbf"].shape)
  38. print("gradients[\"dbi\"][4] =", gradients_tmp["dbi"][4])
  39. print("gradients[\"dbi\"].shape =", gradients_tmp["dbi"].shape)
  40. print("gradients[\"dbc\"][4] =", gradients_tmp["dbc"][4])
  41. print("gradients[\"dbc\"].shape =", gradients_tmp["dbc"].shape)
  42. print("gradients[\"dbo\"][4] =", gradients_tmp["dbo"][4])
  43. print("gradients[\"dbo\"].shape =", gradients_tmp["dbo"].shape)

gradients["dxt"][1][2] = 3.2305591151091875
gradients["dxt"].shape = (3, 10)
gradients["da_prev"][2][3] = -0.06396214197109236
gradients["da_prev"].shape = (5, 10)
gradients["dc_prev"][2][3] = 0.7975220387970015
gradients["dc_prev"].shape = (5, 10)
gradients["dWf"][3][1] = -0.1479548381644968
gradients["dWf"].shape = (5, 8)
gradients["dWi"][1][2] = 1.0574980552259903
gradients["dWi"].shape = (5, 8)
gradients["dWc"][3][1] = 2.3045621636876668
gradients["dWc"].shape = (5, 8)
gradients["dWo"][1][2] = 0.3313115952892109
gradients["dWo"].shape = (5, 8)
gradients["dbf"][4] = [0.18864637]
gradients["dbf"].shape = (5, 1)
gradients["dbi"][4] = [-0.40142491]
gradients["dbi"].shape = (5, 1)
gradients["dbc"][4] = [0.25587763]
gradients["dbc"].shape = (5, 1)
gradients["dbo"][4] = [0.13893342]
gradients["dbo"].shape = (5, 1) 

LSTM反向传播
  1. # UNGRADED FUNCTION: lstm_backward
  2.  
  3. def lstm_backward(da, caches):
  4.  
  5.     # Retrieve values from the first cache (t=1) of caches.
  6.     (caches, x) = caches
  7.     (a1, c1, a0, c0, f1, i1, cc1, o1, x1, parameters) = caches[0]
  8.  
  9.     # Retrieve dimensions from da's and x1's shapes (≈2 lines)
  10.     n_a, m, T_x = da.shape
  11.     n_x, m = x1.shape
  12.  
  13.     # initialize the gradients with the right sizes (≈12 lines)
  14.     dx = np.zeros([n_x, m, T_x])
  15.     da0 = np.zeros([n_a, m])
  16.     da_prevt = np.zeros([n_a, m])
  17.     dc_prevt = np.zeros([n_a, m])
  18.     dWf = np.zeros([n_a, n_a + n_x])
  19.     dWi = np.zeros([n_a, n_a + n_x])
  20.     dWc = np.zeros([n_a, n_a + n_x])
  21.     dWo = np.zeros([n_a, n_a + n_x])
  22.     dbf = np.zeros([n_a, 1])
  23.     dbi = np.zeros([n_a, 1])
  24.     dbc = np.zeros([n_a, 1])
  25.     dbo = np.zeros([n_a, 1])
  26.  
  27.     # loop back over the whole sequence
  28.     for t in reversed(range(T_x)):
  29.         # Compute all gradients using lstm_cell_backward
  30.         gradients = lstm_cell_backward(da[:, :, t] + da_prevt, dc_prevt, caches[t])
  31.         # Store or add the gradient to the parameters' previous step's gradient
  32.         da_prevt = gradients["da_prev"]
  33.         dc_prevt = gradients["dc_prev"]
  34.         dx[:, :, t] = gradients["dxt"]
  35.         dWf = dWf + gradients["dWf"]
  36.         dWi = dWi + gradients["dWi"]
  37.         dWc = dWc + gradients["dWc"]
  38.         dWo = dWo + gradients["dWo"]
  39.         dbf = dbf + gradients["dbf"]
  40.         dbi = dbi + gradients["dbi"]
  41.         dbc = dbc + gradients["dbc"]
  42.         dbo = dbo + gradients["dbo"]
  43.     # Set the first activation's gradient to the backpropagated gradient da_prev.
  44.     da0 = gradients["da_prev"]
  45.  
  46.     # Store the gradients in a python dictionary
  47.     gradients = {"dx": dx, "da0": da0, "dWf": dWf,"dbf": dbf, "dWi": dWi,"dbi": dbi,
  48.                 "dWc": dWc,"dbc": dbc, "dWo": dWo,"dbo": dbo}
  49.  
  50.     return gradients
  1. np.random.seed(1)
  2. x_tmp = np.random.randn(3,10,7)
  3. a0_tmp = np.random.randn(5,10)
  4.  
  5. parameters_tmp = {}
  6. parameters_tmp['Wf'] = np.random.randn(5, 5+3)
  7. parameters_tmp['bf'] = np.random.randn(5,1)
  8. parameters_tmp['Wi'] = np.random.randn(5, 5+3)
  9. parameters_tmp['bi'] = np.random.randn(5,1)
  10. parameters_tmp['Wo'] = np.random.randn(5, 5+3)
  11. parameters_tmp['bo'] = np.random.randn(5,1)
  12. parameters_tmp['Wc'] = np.random.randn(5, 5+3)
  13. parameters_tmp['bc'] = np.random.randn(5,1)
  14. parameters_tmp['Wy'] = np.zeros((2,5))       # unused, but needed for lstm_forward
  15. parameters_tmp['by'] = np.zeros((2,1))       # unused, but needed for lstm_forward
  16.  
  17. a_tmp, y_tmp, c_tmp, caches_tmp = lstm_forward(x_tmp, a0_tmp, parameters_tmp)
  18.  
  19. da_tmp = np.random.randn(5, 10, 4)
  20. gradients_tmp = lstm_backward(da_tmp, caches_tmp)
  21.  
  22. print("gradients[\"dx\"][1][2] =", gradients_tmp["dx"][1][2])
  23. print("gradients[\"dx\"].shape =", gradients_tmp["dx"].shape)
  24. print("gradients[\"da0\"][2][3] =", gradients_tmp["da0"][2][3])
  25. print("gradients[\"da0\"].shape =", gradients_tmp["da0"].shape)
  26. print("gradients[\"dWf\"][3][1] =", gradients_tmp["dWf"][3][1])
  27. print("gradients[\"dWf\"].shape =", gradients_tmp["dWf"].shape)
  28. print("gradients[\"dWi\"][1][2] =", gradients_tmp["dWi"][1][2])
  29. print("gradients[\"dWi\"].shape =", gradients_tmp["dWi"].shape)
  30. print("gradients[\"dWc\"][3][1] =", gradients_tmp["dWc"][3][1])
  31. print("gradients[\"dWc\"].shape =", gradients_tmp["dWc"].shape)
  32. print("gradients[\"dWo\"][1][2] =", gradients_tmp["dWo"][1][2])
  33. print("gradients[\"dWo\"].shape =", gradients_tmp["dWo"].shape)
  34. print("gradients[\"dbf\"][4] =", gradients_tmp["dbf"][4])
  35. print("gradients[\"dbf\"].shape =", gradients_tmp["dbf"].shape)
  36. print("gradients[\"dbi\"][4] =", gradients_tmp["dbi"][4])
  37. print("gradients[\"dbi\"].shape =", gradients_tmp["dbi"].shape)
  38. print("gradients[\"dbc\"][4] =", gradients_tmp["dbc"][4])
  39. print("gradients[\"dbc\"].shape =", gradients_tmp["dbc"].shape)
  40. print("gradients[\"dbo\"][4] =", gradients_tmp["dbo"][4])
  41. print("gradients[\"dbo\"].shape =", gradients_tmp["dbo"].shape)

gradients["dx"][1][2] = [ 0.00218254  0.28205375 -0.48292508 -0.43281115]
gradients["dx"].shape = (3, 10, 4)
gradients["da0"][2][3] = 0.31277031025726026
gradients["da0"].shape = (5, 10)
gradients["dWf"][3][1] = -0.08098023109383463
gradients["dWf"].shape = (5, 8)
gradients["dWi"][1][2] = 0.40512433092981837
gradients["dWi"].shape = (5, 8)
gradients["dWc"][3][1] = -0.07937467355121491
gradients["dWc"].shape = (5, 8)
gradients["dWo"][1][2] = 0.038948775762986956
gradients["dWo"].shape = (5, 8)
gradients["dbf"][4] = [-0.15745657]
gradients["dbf"].shape = (5, 1)
gradients["dbi"][4] = [-0.50848333]
gradients["dbi"].shape = (5, 1)
gradients["dbc"][4] = [-0.42510818]
gradients["dbc"].shape = (5, 1)
gradients["dbo"][4] = [-0.17958196]
gradients["dbo"].shape = (5, 1)

祝贺你完成了这个作业！

现在你明白了循环神经网络是如何工作的！在下一个练习中，您将使用RNN来构建字符级语言模型。到时见！

3 字符级语言模型-恐龙岛

欢迎来到恐龙岛！六千五百万年前，恐龙就存在了，在这次任务中，它们又回来了。

你负责一项特殊的任务：领先的生物学研究人员正在创造新的恐龙品种，并将它们带到地球上，而你的工作是为这些恐龙命名。如果恐龙不喜欢自己的名字，它可能会发狂。所以要明智地选择！

幸运的是，你现在已经具备了一些深度学习能力，你将用它来拯救世界！你的助手收集了他们能找到的所有恐龙名字的列表，并将它们汇编成这个[数据集]（dinosaur .txt）。（请点击前面的链接查看。）要创建新的恐龙名称，您将构建一个字符级语言模型来生成新的名称。您的算法将学习不同的名称模式，并随机生成新名称。希望这个算法能让你和你的团队远离恐龙的愤怒！

当你完成这个作业时，你将能够：

  1. * 存储文本数据，使用RNN进行处理
  2. * 使用RNN构建字符级文本生成模型
  3. * RNN中的新序列样本
  4. * 解释rnn中的梯度消失/爆炸问题
  5. * 应用梯度裁剪作为爆炸梯度的解决方案

首先加载‘ rn_utils ’中为您提供的一些函数。具体来说，你可以访问像‘ rnn_forward ’和‘ rnn_backward ’这样的函数，它们相当于你在前面的赋值中实现的函数。

3.0 导包

  1. import numpy as np
  2. from utils import *
  3. import random
  4. import pprint
  5. import copy

3.1 问题重述

数据集与预处理

运行下面的单元格读取恐龙名称数据集，创建唯一字符列表（如a-z），并计算数据集和词汇表大小。

  1. data = open('dinos.txt', 'r').read()
  2. data= data.lower()
  3. chars = list(set(data))
  4. data_size, vocab_size = len(data), len(chars)
  5. print('There are %d total characters and %d unique characters in your data.' % (data_size, vocab_size))

There are 19909 total characters and 27 unique characters in your data. 

  1. * 字符是a-z（26个字符）加上“\n”（或换行字符）。
  2. * 在此作业中，换行字符“\n”的作用类似于讲座中讨论的“”（或“句子结束”）标记。
  3. 在这里，“\n”表示恐龙名字的结尾，而不是句子的结尾。
  4. * ' char_to_ix '：在下面的单元格中，您将创建一个Python字典（即哈希表），将每个字符映射到从0-26的索引。
  5. * ' ix_to_char '：然后，创建第二个Python字典，将每个索引映射回相应的字符。
  6. - 这将帮助您找出哪个索引对应于softmax层的概率分布输出中的哪个字符。
  1. chars = sorted(chars)
  2. print(chars)

['\n', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

  1. char_to_ix = { ch:i for i,ch in enumerate(chars) }
  2. ix_to_char = { i:ch for i,ch in enumerate(chars) }
  3. pp = pprint.PrettyPrinter(indent=4)
  4. pp.pprint(ix_to_char)

{   0: '\n',
    1: 'a',
    2: 'b',
    3: 'c',
    4: 'd',
    5: 'e',
    6: 'f',
    7: 'g',
    8: 'h',
    9: 'i',
    10: 'j',
    11: 'k',
    12: 'l',
    13: 'm',
    14: 'n',
    15: 'o',
    16: 'p',
    17: 'q',
    18: 'r',
    19: 's',
    20: 't',
    21: 'u',
    22: 'v',
    23: 'w',
    24: 'x',
    25: 'y',
    26: 'z'} 

模型概述

您的模型将具有以下结构：

  1. - 初始化参数
  2. - 执行优化循环
  3. - 前向传播，计算损失函数
  4. - 反向传播计算梯度相对于损失函数
  5. - 剪辑梯度，以避免爆炸梯度
  6. - 使用梯度，用梯度下降更新规则更新参数。
  7. - 返回学习到的参数

3.2 构建模型的小块

在这一部分中，您将构建整个模型的两个重要模块：

1. 渐变裁剪: 避免渐变爆炸

2. 采样：一种用来生成字符的技术

然后，您将应用这两个函数来构建模型。

裁剪优化循环中的梯度

在本节中，您将实现在优化循环中调用的“clip”函数。

  • * 有不同的方法来剪辑渐变。
  • * 你将使用一个简单的元素裁剪过程，在这个过程中，梯度向量的每个元素都被裁剪到某个范围[-N， N]之间。
  • * 例如，如果N=10，则范围为[- 10,10]
  • - 如果梯度向量的任何分量大于10，则设置为10。
  • - 如果梯度向量的任何分量小于-10，则设置为-10。
  • - 如果组件值在-10 ~ 10之间，则保持原值。
在网络遇到“梯度爆炸”问题的情况下，有梯度裁剪和没有梯度裁剪的梯度下降可视化。 
  1. # UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. ### GRADED FUNCTION: clip
  3.  
  4. def clip(gradients, maxValue):
  5.  
  6.     gradients = copy.deepcopy(gradients)
  7.     
  8.     dWaa, dWax, dWya, db, dby = gradients['dWaa'], gradients['dWax'], gradients['dWya'], gradients['db'], gradients['dby']
  9.    
  10.     # Clip to mitigate exploding gradients, loop over [dWax, dWaa, dWya, db, dby]. (≈2 lines)
  11.     for gradient in gradients:
  12.         np.clip(gradients[gradient], -maxValue, maxValue, out = gradients[gradient])
  13.     
  14.     gradients = {"dWaa": dWaa, "dWax": dWax, "dWya": dWya, "db": db, "dby": dby}
  15.     
  16.     return gradients
  1. # Test with a max value of 10
  2. def clip_test(target, mValue):
  3.     print(f"\nGradients for mValue={mValue}")
  4.     np.random.seed(3)
  5.     dWax = np.random.randn(5, 3) * 10
  6.     dWaa = np.random.randn(5, 5) * 10
  7.     dWya = np.random.randn(2, 5) * 10
  8.     db = np.random.randn(5, 1) * 10
  9.     dby = np.random.randn(2, 1) * 10
  10.     gradients = {"dWax": dWax, "dWaa": dWaa, "dWya": dWya, "db": db, "dby": dby}
  11.  
  12.     gradients2 = target(gradients, mValue)
  13.     print("gradients[\"dWaa\"][1][2] =", gradients2["dWaa"][1][2])
  14.     print("gradients[\"dWax\"][3][1] =", gradients2["dWax"][3][1])
  15.     print("gradients[\"dWya\"][1][2] =", gradients2["dWya"][1][2])
  16.     print("gradients[\"db\"][4] =", gradients2["db"][4])
  17.     print("gradients[\"dby\"][1] =", gradients2["dby"][1])
  18.     
  19.     for grad in gradients2.keys():
  20.         valuei = gradients[grad]
  21.         valuef = gradients2[grad]
  22.         mink = np.min(valuef)
  23.         maxk = np.max(valuef)
  24.         assert mink >= -abs(mValue), f"Problem with {grad}. Set a_min to -mValue in the np.clip call"
  25.         assert maxk <= abs(mValue), f"Problem with {grad}.Set a_max to mValue in the np.clip call"
  26.         index_not_clipped = np.logical_and(valuei <= mValue, valuei >= -mValue)
  27.         assert np.all(valuei[index_not_clipped] == valuef[index_not_clipped]), f" Problem with {grad}. Some values that should not have changed, changed during the clipping process."
  28.     
  29.     print("\033[92mAll tests passed!\x1b[0m")
  30.     
  31. clip_test(clip, 10)
  32. clip_test(clip, 5)

Gradients for mValue=10
gradients["dWaa"][1][2] = 10.0
gradients["dWax"][3][1] = -10.0
gradients["dWya"][1][2] = 0.2971381536101662
gradients["db"][4] = [10.]
gradients["dby"][1] = [8.45833407]
All tests passed!

Gradients for mValue=5
gradients["dWaa"][1][2] = 5.0
gradients["dWax"][3][1] = -5.0
gradients["dWya"][1][2] = 0.2971381536101662
gradients["db"][4] = [5.]
gradients["dby"][1] = [5.]
All tests passed!

采样

现在，假设您的模型已经过训练，并且希望生成新的文本（字符）。生成过程如下图所示：

  1. # UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: sample
  3.  
  4. def sample(parameters, char_to_ix, seed):
  5.  
  6.     # Retrieve parameters and relevant shapes from "parameters" dictionary
  7.     Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'], parameters['Wya'], parameters['by'], parameters['b']
  8.     vocab_size = by.shape[0]
  9.     n_a = Waa.shape[1]
  10.  
  11.     # Step 1: Create the a zero vector x that can be used as the one-hot vector 
  12.     # Representing the first character (initializing the sequence generation). (≈1 line)
  13.     x = np.zeros((vocab_size,1))
  14.     # Step 1': Initialize a_prev as zeros (≈1 line)
  15.     a_prev = np.zeros((n_a ,1))
  16.  
  17.     # Create an empty list of indices. This is the list which will contain the list of indices of the characters to generate (≈1 line)
  18.     indices = []
  19.  
  20.     # idx is the index of the one-hot vector x that is set to 1
  21.     # All other positions in x are zero.
  22.     # Initialize idx to -1
  23.     idx = -1
  24.  
  25.     # Loop over time-steps t. At each time-step:
  26.     # Sample a character from a probability distribution 
  27.     # And append its index (`idx`) to the list "indices". 
  28.     # You'll stop if you reach 50 characters 
  29.     # (which should be very unlikely with a well-trained model).
  30.     # Setting the maximum number of characters helps with debugging and prevents infinite loops. 
  31.     counter = 0
  32.     newline_character = char_to_ix['\n']
  33.     
  34.     while (idx != newline_character and counter != 50):
  35.  
  36.         # Step 2: Forward propagate x using the equations (1), (2) and (3)
  37.         a = np.tanh(np.dot(Wax,x) + np.dot(Waa,a_prev) + b)
  38.         z = np.dot(Wya,a) + by
  39.         y = softmax(z)
  40.  
  41.         # For grading purposes
  42.         np.random.seed(counter + seed) 
  43.  
  44.         # Step 3: Sample the index of a character within the vocabulary from the probability distribution y
  45.         # (see additional hints above)
  46.         idx = np.random.choice(range(len(y)), p = np.squeeze(y) )
  47.  
  48.         # Append the index to "indices"
  49.         indices.append(idx)
  50.  
  51.         # Step 4: Overwrite the input x with one that corresponds to the sampled index `idx`.
  52.         # (see additional hints above)
  53.         x = np.zeros((vocab_size,1))
  54.         x[idx] = 1
  55.  
  56.         # Update "a_prev" to be "a"
  57.         a_prev = a
  58.  
  59.         # for grading purposes
  60.         seed += 1
  61.  
  62.         counter +=1
  63.  
  64.     if (counter == 50):
  65.         indices.append(char_to_ix['\n'])
  66.     
  67.     return indices
  1. def sample_test(target):
  2.     np.random.seed(24)
  3.     _, n_a = 20, 100
  4.     Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)
  5.     b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)
  6.     parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "b": b, "by": by}
  7.  
  8.  
  9.     indices = target(parameters, char_to_ix, 0)
  10.     print("Sampling:")
  11.     print("list of sampled indices:\n", indices)
  12.     print("list of sampled characters:\n", [ix_to_char[i] for i in indices])
  13.     
  14.     assert len(indices) < 52, "Indices lenght must be smaller than 52"
  15.     assert indices[-1] == char_to_ix['\n'], "All samples must end with \\n"
  16.     assert min(indices) >= 0 and max(indices) < len(char_to_ix), f"Sampled indexes must be between 0 and len(char_to_ix)={len(char_to_ix)}"
  17.     assert np.allclose(indices[0:6], [23, 16, 26, 26, 24, 3]), "Wrong values"
  18.     
  19.     print("\033[92mAll tests passed!")
  20.  
  21. sample_test(sample)

Sampling:
list of sampled indices:
 [23, 16, 26, 26, 24, 3, 21, 1, 7, 24, 15, 3, 25, 20, 6, 13, 10, 8, 20, 12, 2, 0]
list of sampled characters:
 ['w', 'p', 'z', 'z', 'x', 'c', 'u', 'a', 'g', 'x', 'o', 'c', 'y', 't', 'f', 'm', 'j', 'h', 't', 'l', 'b', '\n']
All tests passed!

你应该记住的：

  1. * 非常大或“爆炸”的梯度更新可能太大，以至于它们在反向支撑期间“超调”了最优值——使训练变得困难
  2. * 剪辑梯度之前更新参数，以避免爆炸梯度
  3. * 抽样是一种可以用来根据概率分布选择下一个字符的索引的技术。
  4. * 开始字符级采样：
  5. * 输入一个零的“虚拟”向量作为默认输入
  6. * 运行一步前向传播以获得?〈1〉（您的第一个字符）和expes〈1〉（以下字符的概率分布）
  7. * 在抽样时，通过使用‘ np.random.choice ’来避免每次给定起始字母都产生相同的结果（并使您的名字更有趣！）

3.3 构建语言模型

是时候构建用于文本生成的字符级语言模型了！

梯度下降法

在本节中，您将实现一个函数，执行随机梯度下降的一步（带有剪切梯度）。你会一次看一个训练样例，所以优化算法是随机梯度下降。

提醒一下，以下是RNN常见优化循环的步骤：

  • - 通过RNN前向传播，计算损失
  • - 通过时间反向传播，计算损失相对于参数的梯度
  • - 剪辑渐变
  • - 使用梯度下降法更新参数
  1. # UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: optimize
  3.  
  4. def optimize(X, Y, a_prev, parameters, learning_rate = 0.01):
  5.     
  6.     # Forward propagate through time (≈1 line)
  7.     loss, cache = rnn_forward(X, Y, a_prev, parameters)
  8.     
  9.     # Backpropagate through time (≈1 line)
  10.     gradients, a = rnn_backward(X, Y, parameters, cache)
  11.     
  12.     # Clip your gradients between -5 (min) and 5 (max) (≈1 line)
  13.     gradients = clip(gradients, 5)
  14.     
  15.     # Update parameters (≈1 line)
  16.     parameters = update_parameters(parameters, gradients, learning_rate)
  17.     
  18.     return loss, gradients, a[len(X)-1]

  1. def optimize_test(target):
  2.     np.random.seed(1)
  3.     vocab_size, n_a = 27, 100
  4.     a_prev = np.random.randn(n_a, 1)
  5.     Wax, Waa, Wya = np.random.randn(n_a, vocab_size), np.random.randn(n_a, n_a), np.random.randn(vocab_size, n_a)
  6.     b, by = np.random.randn(n_a, 1), np.random.randn(vocab_size, 1)
  7.     parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "b": b, "by": by}
  8.     X = [12, 3, 5, 11, 22, 3]
  9.     Y = [4, 14, 11, 22, 25, 26]
  10.     old_parameters = copy.deepcopy(parameters)
  11.     loss, gradients, a_last = target(X, Y, a_prev, parameters, learning_rate = 0.01)
  12.     print("Loss =", loss)
  13.     print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
  14.     print("np.argmax(gradients[\"dWax\"]) =", np.argmax(gradients["dWax"]))
  15.     print("gradients[\"dWya\"][1][2] =", gradients["dWya"][1][2])
  16.     print("gradients[\"db\"][4] =", gradients["db"][4])
  17.     print("gradients[\"dby\"][1] =", gradients["dby"][1])
  18.     print("a_last[4] =", a_last[4])
  19.     
  20.     assert np.isclose(loss, 126.5039757), "Problems with the call of the rnn_forward function"
  21.     for grad in gradients.values():
  22.         assert np.min(grad) >= -5, "Problems in the clip function call"
  23.         assert np.max(grad) <= 5, "Problems in the clip function call"
  24.     assert np.allclose(gradients['dWaa'][1, 2], 0.1947093), "Unexpected gradients. Check the rnn_backward call"
  25.     assert np.allclose(gradients['dWya'][1, 2], -0.007773876), "Unexpected gradients. Check the rnn_backward call"
  26.     assert not np.allclose(parameters['Wya'], old_parameters['Wya']), "parameters were not updated"
  27.     
  28.     print("\033[92mAll tests passed!")
  29.  
  30. optimize_test(optimize)

Loss = 126.50397572165382
gradients["dWaa"][1][2] = 0.19470931534716163
np.argmax(gradients["dWax"]) = 93
gradients["dWya"][1][2] = -0.007773876032002922
gradients["db"][4] = [-0.06809825]
gradients["dby"][1] = [0.01538192]
a_last[4] = [-1.]
All tests passed!

模型训练

* 给定恐龙名称数据集，您将使用数据集的每一行（一个名称）作为一个训练示例。

* 每2000步的随机梯度下降，你将抽样几个随机选择的名字，看看算法是如何做的。

  1. # UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: model
  3.  
  4. def model(data_x, ix_to_char, char_to_ix, num_iterations = 35000, n_a = 50, dino_names = 7, vocab_size = 27, verbose = False):
  5.     
  6.     # Retrieve n_x and n_y from vocab_size
  7.     n_x, n_y = vocab_size, vocab_size
  8.     
  9.     # Initialize parameters
  10.     parameters = initialize_parameters(n_a, n_x, n_y)
  11.     
  12.     # Initialize loss (this is required because we want to smooth our loss)
  13.     loss = get_initial_loss(vocab_size, dino_names)
  14.     
  15.     # Build list of all dinosaur names (training examples).
  16.     examples = [x.strip() for x in data_x]
  17.     
  18.     # Shuffle list of all dinosaur names
  19.     np.random.seed(0)
  20.     np.random.shuffle(examples)
  21.     
  22.     # Initialize the hidden state of your LSTM
  23.     a_prev = np.zeros((n_a, 1))
  24.     
  25.     # for grading purposes
  26.     last_dino_name = "abc"
  27.     
  28.     # Optimization loop
  29.     for j in range(num_iterations):
  30.         
  31.         # Set the index `idx` (see instructions above)
  32.         idx = j%len(examples)
  33.         
  34.         # Set the input X (see instructions above)
  35.         single_example_chars = examples[idx]
  36.         single_example_ix = [char_to_ix[c] for c in single_example_chars]
  37.  
  38.         # if X[t] == None, we just have x[t]=0. This is used to set the input for the first timestep to the zero vector. 
  39.         X = [None] + single_example_ix
  40.         
  41.         # Set the labels Y (see instructions above)
  42.         # The goal is to train the RNN to predict the next letter in the name
  43.         # So the labels are the list of characters that are one time-step ahead of the characters in the input X
  44.         Y = X[1:] 
  45.         # The RNN should predict a newline at the last letter, so add ix_newline to the end of the labels
  46.         ix_newline = [char_to_ix["\n"]]
  47.         Y = Y + ix_newline
  48.  
  49.         # Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters
  50.         # Choose a learning rate of 0.01
  51.         curr_loss, gradients, a_prev = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)
  52.         
  53.         # debug statements to aid in correctly forming X, Y
  54.         if verbose and j in [0, len(examples) -1, len(examples)]:
  55.             print("j = " , j, "idx = ", idx,) 
  56.         if verbose and j in [0]:
  57.             #print("single_example =", single_example)
  58.             print("single_example_chars", single_example_chars)
  59.             print("single_example_ix", single_example_ix)
  60.             print(" X = ", X, "\n", "Y =       ", Y, "\n")
  61.         
  62.         # Use a latency trick to keep the loss smooth. It happens here to accelerate the training.
  63.         loss = smooth(loss, curr_loss)
  64.  
  65.         # Every 1000 Iteration, generate "n" characters thanks to sample() to check if the model is learning properly
  66.         if j % 1000 == 0:
  67.             
  68.             print('Iteration: %d, Loss: %f' % (j, loss) + '\n')
  69.             
  70.             # The number of dinosaur names to print
  71.             seed = 0
  72.             for name in range(dino_names):
  73.                 
  74.                 # Sample indices and print them
  75.                 sampled_indices = sample(parameters, char_to_ix, seed)
  76.                 last_dino_name = get_sample(sampled_indices, ix_to_char)
  77.                 print(last_dino_name.replace('\n', ''))
  78.                 
  79.                 seed += 1  # To get the same result (for grading purposes), increment the seed by one. 
  80.       
  81.             print('\n')
  82.         
  83.     return parameters, last_dino_name

当您运行下面的单元格时，您应该看到您的模型在第一次迭代时输出看起来随机的字符。经过几千次迭代后，您的模型应该学会生成看起来合理的名称。

  1. parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
  2.  
  3. assert last_name == 'Trodonosaurus\n', "Wrong expected output"
  4. print("\033[92mAll tests passed!")
  1. j =  0 idx =  0
  2. single_example_chars turiasaurus
  3. single_example_ix [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19]
  4.  X =  [None, 20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19] 
  5.  Y =        [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19, 0] 
  6.  
  7. Iteration: 0, Loss: 23.087336
  8.  
  9. Nkzxwtdmfqoeyhsqwasjkjvu
  10. Kneb
  11. Kzxwtdmfqoeyhsqwasjkjvu
  12. Neb
  13. Zxwtdmfqoeyhsqwasjkjvu
  14. Eb
  15. Xwtdmfqoeyhsqwasjkjvu
  16.  
  17.  
  18. Iteration: 1000, Loss: 28.712699
  19.  
  20. Nivusahidoraveros
  21. Ioia
  22. Iwtroeoirtaurusabrngeseaosawgeanaitafeaolaeratohop
  23. Nac
  24. Xtroeoirtaurusabrngeseaosawgeanaitafeaolaeratohopr
  25. Ca
  26. Tseeohnnaveros
  27.  
  28.  
  29. j =  1535 idx =  1535
  30. j =  1536 idx =  0
  31. Iteration: 2000, Loss: 27.884160
  32.  
  33. Liusskeomnolxeros
  34. Hmdaairus
  35. Hytroligoraurus
  36. Lecalosapaus
  37. Xusicikoraurus
  38. Abalpsamantisaurus
  39. Tpraneronxeros
  40.  
  41.  
  42. Iteration: 3000, Loss: 26.863598
  43.  
  44. Niusos
  45. Infa
  46. Iusrtendor
  47. Nda
  48. Wtrololos
  49. Ca
  50. Tps
  51.  
  52.  
  53. Iteration: 4000, Loss: 25.901815
  54.  
  55. Mivrosaurus
  56. Inee
  57. Ivtroplisaurus
  58. Mbaaisaurus
  59. Wusichisaurus
  60. Cabaselachus
  61. Toraperlethosdarenitochusthiamamumamaon
  62.  
  63.  
  64. Iteration: 5000, Loss: 25.290275
  65.  
  66. Ngyusedonis
  67. Klecagropechus
  68. Lytosaurus
  69. Necagropechusangotmeeycerum
  70. Xuskangosaurus
  71. Da
  72. Tosaurus
  73.  
  74.  
  75. Iteration: 6000, Loss: 24.608779
  76.  
  77. Onwusceomosaurus
  78. Lieeaerosaurus
  79. Lxussaurus
  80. Oma
  81. Xusteonosaurus
  82. Eeahosaurus
  83. Toreonosaurus
  84.  
  85.  
  86. Iteration: 7000, Loss: 24.425330
  87.  
  88. Ngytromiasaurus
  89. Ingabcosaurus
  90. Kyusichiropurusanrasauraptous
  91. Necamithachusidinysaus
  92. Yusodon
  93. Caaesaurus
  94. Tosaurus
  95.  
  96.  
  97. Iteration: 8000, Loss: 24.070350
  98.  
  99. Onxusichepriuon
  100. Kilabersaurus
  101. Lutrodon
  102. Omaaerosaurus
  103. Xutrcheps
  104. Edaksoje
  105. Trodiktonus
  106.  
  107.  
  108. Iteration: 9000, Loss: 23.730944
  109.  
  110. Onyusaurus
  111. Klecanotal
  112. Kyuspang
  113. Ogaacosaurus
  114. Xutrasaurus
  115. Dabcosaurus
  116. Troching
  117.  
  118.  
  119. Iteration: 10000, Loss: 23.844446
  120.  
  121. Onyusaurus
  122. Klecalosaurus
  123. Lustodon
  124. Ola
  125. Xusodonia
  126. Eeaeosaurus
  127. Troceosaurus
  128.  
  129.  
  130. Iteration: 11000, Loss: 23.581901
  131.  
  132. Leutosaurus
  133. Inda
  134. Itrtoplerosherotarangos
  135. Lecalosaurus
  136. Xutogolosaurus
  137. Babator
  138. Trodonosaurus
  139.  
  140.  
  141. Iteration: 12000, Loss: 23.291971
  142.  
  143. Onyxosaurus
  144. Kica
  145. Lustrepiosaurus
  146. Olaagrraiansaurus
  147. Yuspangosaurus
  148. Eealosaurus
  149. Trognesaurus
  150.  
  151.  
  152. Iteration: 13000, Loss: 23.547611
  153.  
  154. Nixrosaurus
  155. Indabcosaurus
  156. Jystolong
  157. Necalosaurus
  158. Yuspangosaurus
  159. Daagosaurus
  160. Usndicirax
  161.  
  162.  
  163. Iteration: 14000, Loss: 23.382338
  164.  
  165. Meutromodromurus
  166. Inda
  167. Iutroinatorsaurus
  168. Maca
  169. Yusteratoptititan
  170. Ca
  171. Troclosaurus
  172.  
  173.  
  174. Iteration: 15000, Loss: 23.049756
  175.  
  176. Phyusaurus
  177. Lidaa
  178. Lustraodon
  179. Padaeron
  180. Yuspchinnaugus
  181. Edalosaurus
  182. Trodon
  183.  
  184.  
  185. Iteration: 16000, Loss: 23.282946
  186.  
  187. Mdyusaurus
  188. Indaacosaupisaurus
  189. Justolong
  190. Maca
  191. Yuspandosaurus
  192. Cabaspadantes
  193. Trodon
  194.  
  195.  
  196. Iteration: 17000, Loss: 23.156690
  197.  
  198. Ootstrethosaurus
  199. Jica
  200. Kustonagor
  201. Ola
  202. Yustanchohugrosaurus
  203. Eeagosaurus
  204. Trpenesaurus
  205.  
  206.  
  207. Iteration: 18000, Loss: 22.850813
  208.  
  209. Phyusaurus
  210. Meja
  211. Mystoosaurus
  212. Pegamosaurus
  213. Yusmaphosaurus
  214. Eiahosaurus
  215. Trolonosaurus
  216.  
  217.  
  218. Iteration: 19000, Loss: 23.046266
  219.  
  220. Opusaurus
  221. Kola
  222. Lustolonis
  223. Ola
  224. Yustanisaurus
  225. Eiahosaurus
  226. Trofonosaurus
  227.  
  228.  
  229. Iteration: 20000, Loss: 22.929326
  230.  
  231. Nlyusaurus
  232. Logbalosaurus
  233. Lvuslangosaurus
  234. Necalosaurus
  235. Ytrrangosaurus
  236. Ekairus
  237. Troenesaurus
  238.  
  239.  
  240. Iteration: 21000, Loss: 22.672015
  241.  
  242. Phyusaurus
  243. Loeia
  244. Lyutorosaurus
  245. Pacalosaurus
  246. Yusodon
  247. Egaerosaurus
  248. Troholosaurus
  249.  
  250.  
  251. Iteration: 22000, Loss: 22.760728
  252.  
  253. Onvusaroglolonoshareimus
  254. Llecaerona
  255. Myrrocephoeurus
  256. Pedacosaurus
  257. Ytrodonosaurus
  258. Eiadosaurus
  259. Trodonosaurus
  260.  
  261.  
  262. All tests passed!

3.4 结论

你可以看到，在训练结束时，你的算法已经开始生成合理的恐龙名称。一开始，它是随机生成角色，但到最后，你可以开始看到恐龙的名字有很酷的结尾。请随意运行算法更长时间，并使用超参数，看看是否可以获得更好的结果！我们的实现生成了一些非常酷的名字，比如“maconucon”、“marloralus”和“macingsersaurus”。希望你的模型也了解到恐龙的名字往往以“saurus”、“don”、“aura”、“tor”等结尾。

如果你的模型产生了一些不酷的名字，不要完全责怪模型——并不是所有真正的恐龙名字听起来都很酷。（例如，“dromaeosauroides”是一个真实的恐龙名称，并且在训练集中。）但这个模型应该会给你一组候选对象，你可以从中选择最酷的！

这个任务使用了一个相对较小的数据集，这样你就可以在CPU上快速训练一个RNN。训练一个英语语言模型需要更大的数据集，通常需要更多的计算，并且可以在gpu上运行几个小时。我们用了很长一段时间的恐龙名字，到目前为止我们最喜欢的名字是伟大的，凶猛的，不败的：**芒果龙**！

您已经完成了本笔记本的分级部分，并创建了一个工作语言模型！很棒的工作。

到目前为止，你已经：

  1. * 使用RNN存储文本数据进行处理
  2. * 建立字符级文本生成模型
  3. * 探索了rnn中的梯度消失/爆炸问题
  4. * 应用梯度裁剪，以避免爆炸梯度

你也有希望生成一些足够酷的恐龙名字，既能取悦你，又能避免恐龙的愤怒。

3.5 像莎士比亚一样写作

与字符级文本生成类似（但更复杂）的任务是生成莎士比亚诗歌。你可以使用莎士比亚的诗集，而不是从恐龙名字的数据集中学习。使用LSTM单元，您可以学习跨越文本中许多字符的长期依赖关系。，一个角色出现在序列的某个地方，可能会影响到序列后面应该出现的不同角色。这些长期的依赖关系对于恐龙的名字来说就不那么重要了，因为恐龙的名字都很短。

下面，您可以使用Keras实现莎士比亚诗歌生成器。运行下面的单元格来加载所需的包和模型。这可能需要几分钟。

  1. from __future__ import print_function
  2. from tensorflow.keras.callbacks import LambdaCallback
  3. from tensorflow.keras.models import Model, load_model, Sequential
  4. from tensorflow.keras.layers import Dense, Activation, Dropout, Input, Masking
  5. from tensorflow.keras.layers import LSTM
  6. from tensorflow.keras.utils import get_file
  7. from tensorflow.keras.preprocessing.sequence import pad_sequences
  8. from shakespeare_utils import *
  9. import sys
  10. import io
  11. print("\nModel & Data Loaded\n")

为了节省您的时间，我们已经对一个名为“[十四行诗](shakespeare.txt)”的莎士比亚诗歌集进行了约1000个epoch的模型训练。

让我们再训练一个epoch的模型。当它完成epoch的训练时（这也需要几分钟），您可以运行‘ generate_output ’，它将提示您输入（' < ' 40个字符）。这首诗将从你的句子开始，你的RNN莎士比亚将为你完成这首诗的其余部分！例如，试试“Forsooth this maketh no sense”（没有引号！）。根据您是否在末尾包含空格，您的结果也可能有所不同，因此请尝试两种方式，并尝试其他输入。

  1. print_callback = LambdaCallback(on_epoch_end=on_epoch_end)
  2.  
  3. model.fit(x, y, batch_size=128, epochs=1, callbacks=[print_callback])

246/246 [==============================] - 128s 505ms/step - loss: 2.5351 

  1. # Run this cell to try with different inputs without having to re-train the model 
  2. generate_output()

Input:Forsooth this maketh no sense 

Here is your poem: 

Forsooth this maketh no sense,
live of love mased beeuted liot poftery,
which espunty eye but my didelven heart.
'owt thos eyes maimed woeds inled and to dele.
to all i sane eye table me,
thin eid the styout befooe with muting,
and one, ar the kthat than on bother bate things lofe,
a difl a live convedamed, will hif daker,
of love onti swast, fore hath unhir storsed,
assange i primacan defeding be iy more,
who show both is hi

祝贺你完成了这本笔记本！

RNN莎士比亚模型和你为恐龙命名建立的模型非常相似。唯一的主要区别是：

  1. - lstm代替基本RNN来捕获更远距离的依赖关系
  2. - 模型是一个更深的，堆叠的LSTM模型（2层）
  3. - 使用Keras代替Python来简化代码

4 用LSTM网络即兴演奏爵士独奏

欢迎来到本周最后的编程作业！在本笔记本中，您将实现一个使用LSTM生成音乐的模型。最后，你甚至可以听自己的音乐！

在本次作业结束时，你将能够：

- 将LSTM应用于音乐生成任务

- 用深度学习生成你自己的爵士音乐

- 使用灵活的Functional API来创建复杂的模型

这会很有趣的。我们开始吧！

4.0 导包

  1. import IPython
  2. import sys
  3. import matplotlib.pyplot as plt
  4. import numpy as np
  5. import tensorflow as tf
  6.  
  7. from music21 import *
  8. from grammar import *
  9. from qa import *
  10. from preprocess import * 
  11. from music_utils import *
  12. from data_utils import *
  13. from outputs import *
  14. from test_utils import *
  15.  
  16. from tensorflow.keras.layers import Dense, Activation, Dropout, Input, LSTM, Reshape, Lambda, RepeatVector
  17. from tensorflow.keras.models import Model
  18. from tensorflow.keras.optimizers import Adam
  19. from tensorflow.keras.utils import to_categorical
  20. physical_gpus = tf.config.list_physical_devices("GPU")
  21. tf.config.experimental.set_memory_growth(physical_gpus[0], True)
  22. logical_gpus = tf.config.list_logical_devices("GPU")

4.1 问题重述

你想为朋友的生日特别创作一首爵士音乐。然而，你不知道如何演奏乐器，也不知道如何作曲。幸运的是，您了解深度学习，并将使用LSTM网络解决这个问题！

你将训练一个网络来生成新的爵士独奏，其风格代表了一组已完成的作品。??

数据集

首先，您将在爵士音乐语料库上训练您的算法。运行下面的单元格来收听训练集中的音频片段：

IPython.display.Audio('./data/30s_seq.wav')

音乐数据的预处理已经处理好了，对于本笔记本来说，这意味着它已经以音乐“值”的形式呈现。

什么是音乐的“值”？

你可以把每个“值”想象成一个音符，它包括一个音高和一个持续时间。例如，如果您按下一个特定的钢琴键0.5秒，那么您刚刚演奏了一个音符。在音乐理论中，“值”实际上比这更复杂——具体来说，它还捕获了同时演奏多个音符所需的信息。例如，在演奏一段音乐时，你可能同时按下两个钢琴键（同时演奏多个音符会产生所谓的“和弦”）。但是你们不需要担心这个作业中音乐理论的细节。

音乐是一系列的价值观

* 为了这个作业的目的，你所需要知道的是你将获得一个值的数据集，并将使用RNN模型来生成值的序列。

* 你的音乐生成系统将使用90个独特的值。

  1. X, Y, n_values, indices_values, chords = load_music_utils("data/original_metheny.mid")
  2. print('number of training examples:', X.shape[0])
  3. print('Tx (length of sequence):', X.shape[1])
  4. print('total # of unique values:', n_values)
  5. print('shape of X:', X.shape)
  6. print('Shape of Y:', Y.shape)

number of training examples: 60
Tx (length of sequence): 30
total # of unique values: 90
shape of X: (60, 30, 90)
Shape of Y: (30, 60, 90)

模型概述

下面是您将使用的模型的体系结构。它类似于Dinosaurus模型，只不过您将在Keras中实现它。

在第2部分中，您将训练一个模型，该模型预测下一个音符的风格与它所训练的爵士音乐相似。训练包含在模型的权重和偏差中。

然后，在第3节中，您将在预测一系列音符的新模型中使用这些权重和偏差，并使用前一个音符来预测下一个音符。

4.2 构造模型

现在，你将建立并训练一个学习音乐模式的模型。

* 模型接受形状$(m, T_x, 90)$的输入X和形状$(T_y, m, 90)$的标签Y。

* 您将使用具有$n_{a} = 64$维的隐藏状态的LSTM。

  1. # number of dimensions for the hidden state of each LSTM cell.
  2. n_a = 64 
  1. n_values = 90 # number of music values
  2. reshaper = Reshape((1, n_values))                  # Used in Step 2.B of djmodel(), below
  3. LSTM_cell = LSTM(n_a, return_state = True)         # Used in Step 2.C
  4. densor = Dense(n_values, activation='softmax')     # Used in Step 2.D

  1. # UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: djmodel
  3.  
  4. def djmodel(Tx, LSTM_cell, densor, reshaper):
  5.     # Get the shape of input values
  6.     n_values = densor.units
  7.     
  8.     # Get the number of the hidden state vector
  9.     n_a = LSTM_cell.units
  10.     
  11.     # Define the input layer and specify the shape
  12.     X = Input(shape=(Tx, n_values)) 
  13.     
  14.     # Define the initial hidden state a0 and initial cell state c0
  15.     # using `Input`
  16.     a0 = Input(shape=(n_a,), name='a0')
  17.     c0 = Input(shape=(n_a,), name='c0')
  18.     a = a0
  19.     c = c0
  20.     # Step 1: Create empty list to append the outputs while you iterate (≈1 line)
  21.     outputs = []
  22.     
  23.     # Step 2: Loop from 0 to Tx
  24.     for t in range(Tx):
  25.         
  26.         # Step 2.A: select the "t"th time step vector from X. 
  27.         x = X[:,t,:]
  28.         # Step 2.B: Use reshaper to reshape x to be (1, n_values) (≈1 line)
  29.         x = reshaper(x)
  30.         # Step 2.C: Perform one step of the LSTM_cell
  31.         a, _, c = LSTM_cell(x, initial_state=[a, c])
  32.         # Step 2.D: Apply densor to the hidden state output of LSTM_Cell
  33.         out = densor(a)
  34.         # Step 2.E: add the output to "outputs"
  35.         outputs.append(out)
  36.         
  37.     # Step 3: Create model instance
  38.     model = Model(inputs=[X, a0, c0], outputs=outputs)
  39.     
  40.     return model
创建模型对象

* 运行以下单元格来定义您的模型。

* 我们将使用“Tx=30”。

* 此单元格可能需要几秒钟才能运行。

model = djmodel(Tx=30, LSTM_cell=LSTM_cell, densor=densor, reshaper=reshaper)
  1. # UNIT TEST
  2. output = summary(model) 
  1. # Check your model
  2. model.summary()
  1. Model: "model"
  2. __________________________________________________________________________________________________
  3. Layer (type)                    Output Shape         Param #     Connected to                     
  4. ==================================================================================================
  5. input_1 (InputLayer)            [(None, 30, 90)]     0                                            
  6. __________________________________________________________________________________________________
  7. tf.__operators__.getitem (Slici (None, 90)           0           input_1[0][0]                    
  8. __________________________________________________________________________________________________
  9. reshape (Reshape)               (None, 1, 90)        0           tf.__operators__.getitem[0][0]   
  10.                                                                  tf.__operators__.getitem_1[0][0] 
  11.                                                                  tf.__operators__.getitem_2[0][0] 
  12.                                                                  tf.__operators__.getitem_3[0][0] 
  13.                                                                  tf.__operators__.getitem_4[0][0] 
  14.                                                                  tf.__operators__.getitem_5[0][0] 
  15.                                                                  tf.__operators__.getitem_6[0][0] 
  16.                                                                  tf.__operators__.getitem_7[0][0] 
  17.                                                                  tf.__operators__.getitem_8[0][0] 
  18.                                                                  tf.__operators__.getitem_9[0][0] 
  19.                                                                  tf.__operators__.getitem_10[0][0]
  20.                                                                  tf.__operators__.getitem_11[0][0]
  21.                                                                  tf.__operators__.getitem_12[0][0]
  22.                                                                  tf.__operators__.getitem_13[0][0]
  23.                                                                  tf.__operators__.getitem_14[0][0]
  24.                                                                  tf.__operators__.getitem_15[0][0]
  25.                                                                  tf.__operators__.getitem_16[0][0]
  26.                                                                  tf.__operators__.getitem_17[0][0]
  27.                                                                  tf.__operators__.getitem_18[0][0]
  28.                                                                  tf.__operators__.getitem_19[0][0]
  29.                                                                  tf.__operators__.getitem_20[0][0]
  30.                                                                  tf.__operators__.getitem_21[0][0]
  31.                                                                  tf.__operators__.getitem_22[0][0]
  32.                                                                  tf.__operators__.getitem_23[0][0]
  33.                                                                  tf.__operators__.getitem_24[0][0]
  34.                                                                  tf.__operators__.getitem_25[0][0]
  35.                                                                  tf.__operators__.getitem_26[0][0]
  36.                                                                  tf.__operators__.getitem_27[0][0]
  37.                                                                  tf.__operators__.getitem_28[0][0]
  38.                                                                  tf.__operators__.getitem_29[0][0]
  39. __________________________________________________________________________________________________
  40. a0 (InputLayer)                 [(None, 64)]         0                                            
  41. __________________________________________________________________________________________________
  42. c0 (InputLayer)                 [(None, 64)]         0                                            
  43. __________________________________________________________________________________________________
  44. tf.__operators__.getitem_1 (Sli (None, 90)           0           input_1[0][0]                    
  45. __________________________________________________________________________________________________
  46. lstm (LSTM)                     [(None, 64), (None,  39680       reshape[0][0]                    
  47.                                                                  a0[0][0]                         
  48.                                                                  c0[0][0]                         
  49.                                                                  reshape[1][0]                    
  50.                                                                  lstm[0][0]                       
  51.                                                                  lstm[0][2]                       
  52.                                                                  reshape[2][0]                    
  53.                                                                  lstm[1][0]                       
  54.                                                                  lstm[1][2]                       
  55.                                                                  reshape[3][0]                    
  56.                                                                  lstm[2][0]                       
  57.                                                                  lstm[2][2]                       
  58.                                                                  reshape[4][0]                    
  59.                                                                  lstm[3][0]                       
  60.                                                                  lstm[3][2]                       
  61.                                                                  reshape[5][0]                    
  62.                                                                  lstm[4][0]                       
  63.                                                                  lstm[4][2]                       
  64.                                                                  reshape[6][0]                    
  65.                                                                  lstm[5][0]                       
  66.                                                                  lstm[5][2]                       
  67.                                                                  reshape[7][0]                    
  68.                                                                  lstm[6][0]                       
  69.                                                                  lstm[6][2]                       
  70.                                                                  reshape[8][0]                    
  71.                                                                  lstm[7][0]                       
  72.                                                                  lstm[7][2]                       
  73.                                                                  reshape[9][0]                    
  74.                                                                  lstm[8][0]                       
  75.                                                                  lstm[8][2]                       
  76.                                                                  reshape[10][0]                   
  77.                                                                  lstm[9][0]                       
  78.                                                                  lstm[9][2]                       
  79.                                                                  reshape[11][0]                   
  80.                                                                  lstm[10][0]                      
  81.                                                                  lstm[10][2]                      
  82.                                                                  reshape[12][0]                   
  83.                                                                  lstm[11][0]                      
  84.                                                                  lstm[11][2]                      
  85.                                                                  reshape[13][0]                   
  86.                                                                  lstm[12][0]                      
  87.                                                                  lstm[12][2]                      
  88.                                                                  reshape[14][0]                   
  89.                                                                  lstm[13][0]                      
  90.                                                                  lstm[13][2]                      
  91.                                                                  reshape[15][0]                   
  92.                                                                  lstm[14][0]                      
  93.                                                                  lstm[14][2]                      
  94.                                                                  reshape[16][0]                   
  95.                                                                  lstm[15][0]                      
  96.                                                                  lstm[15][2]                      
  97.                                                                  reshape[17][0]                   
  98.                                                                  lstm[16][0]                      
  99.                                                                  lstm[16][2]                      
  100.                                                                  reshape[18][0]                   
  101.                                                                  lstm[17][0]                      
  102.                                                                  lstm[17][2]                      
  103.                                                                  reshape[19][0]                   
  104.                                                                  lstm[18][0]                      
  105.                                                                  lstm[18][2]                      
  106.                                                                  reshape[20][0]                   
  107.                                                                  lstm[19][0]                      
  108.                                                                  lstm[19][2]                      
  109.                                                                  reshape[21][0]                   
  110.                                                                  lstm[20][0]                      
  111.                                                                  lstm[20][2]                      
  112.                                                                  reshape[22][0]                   
  113.                                                                  lstm[21][0]                      
  114.                                                                  lstm[21][2]                      
  115.                                                                  reshape[23][0]                   
  116.                                                                  lstm[22][0]                      
  117.                                                                  lstm[22][2]                      
  118.                                                                  reshape[24][0]                   
  119.                                                                  lstm[23][0]                      
  120.                                                                  lstm[23][2]                      
  121.                                                                  reshape[25][0]                   
  122.                                                                  lstm[24][0]                      
  123.                                                                  lstm[24][2]                      
  124.                                                                  reshape[26][0]                   
  125.                                                                  lstm[25][0]                      
  126.                                                                  lstm[25][2]                      
  127.                                                                  reshape[27][0]                   
  128.                                                                  lstm[26][0]                      
  129.                                                                  lstm[26][2]                      
  130.                                                                  reshape[28][0]                   
  131.                                                                  lstm[27][0]                      
  132.                                                                  lstm[27][2]                      
  133.                                                                  reshape[29][0]                   
  134.                                                                  lstm[28][0]                      
  135.                                                                  lstm[28][2]                      
  136. __________________________________________________________________________________________________
  137. tf.__operators__.getitem_2 (Sli (None, 90)           0           input_1[0][0]                    
  138. __________________________________________________________________________________________________
  139. tf.__operators__.getitem_3 (Sli (None, 90)           0           input_1[0][0]                    
  140. __________________________________________________________________________________________________
  141. tf.__operators__.getitem_4 (Sli (None, 90)           0           input_1[0][0]                    
  142. __________________________________________________________________________________________________
  143. tf.__operators__.getitem_5 (Sli (None, 90)           0           input_1[0][0]                    
  144. __________________________________________________________________________________________________
  145. tf.__operators__.getitem_6 (Sli (None, 90)           0           input_1[0][0]                    
  146. __________________________________________________________________________________________________
  147. tf.__operators__.getitem_7 (Sli (None, 90)           0           input_1[0][0]                    
  148. __________________________________________________________________________________________________
  149. tf.__operators__.getitem_8 (Sli (None, 90)           0           input_1[0][0]                    
  150. __________________________________________________________________________________________________
  151. tf.__operators__.getitem_9 (Sli (None, 90)           0           input_1[0][0]                    
  152. __________________________________________________________________________________________________
  153. tf.__operators__.getitem_10 (Sl (None, 90)           0           input_1[0][0]                    
  154. __________________________________________________________________________________________________
  155. tf.__operators__.getitem_11 (Sl (None, 90)           0           input_1[0][0]                    
  156. __________________________________________________________________________________________________
  157. tf.__operators__.getitem_12 (Sl (None, 90)           0           input_1[0][0]                    
  158. __________________________________________________________________________________________________
  159. tf.__operators__.getitem_13 (Sl (None, 90)           0           input_1[0][0]                    
  160. __________________________________________________________________________________________________
  161. tf.__operators__.getitem_14 (Sl (None, 90)           0           input_1[0][0]                    
  162. __________________________________________________________________________________________________
  163. tf.__operators__.getitem_15 (Sl (None, 90)           0           input_1[0][0]                    
  164. __________________________________________________________________________________________________
  165. tf.__operators__.getitem_16 (Sl (None, 90)           0           input_1[0][0]                    
  166. __________________________________________________________________________________________________
  167. tf.__operators__.getitem_17 (Sl (None, 90)           0           input_1[0][0]                    
  168. __________________________________________________________________________________________________
  169. tf.__operators__.getitem_18 (Sl (None, 90)           0           input_1[0][0]                    
  170. __________________________________________________________________________________________________
  171. tf.__operators__.getitem_19 (Sl (None, 90)           0           input_1[0][0]                    
  172. __________________________________________________________________________________________________
  173. tf.__operators__.getitem_20 (Sl (None, 90)           0           input_1[0][0]                    
  174. __________________________________________________________________________________________________
  175. tf.__operators__.getitem_21 (Sl (None, 90)           0           input_1[0][0]                    
  176. __________________________________________________________________________________________________
  177. tf.__operators__.getitem_22 (Sl (None, 90)           0           input_1[0][0]                    
  178. __________________________________________________________________________________________________
  179. tf.__operators__.getitem_23 (Sl (None, 90)           0           input_1[0][0]                    
  180. __________________________________________________________________________________________________
  181. tf.__operators__.getitem_24 (Sl (None, 90)           0           input_1[0][0]                    
  182. __________________________________________________________________________________________________
  183. tf.__operators__.getitem_25 (Sl (None, 90)           0           input_1[0][0]                    
  184. __________________________________________________________________________________________________
  185. tf.__operators__.getitem_26 (Sl (None, 90)           0           input_1[0][0]                    
  186. __________________________________________________________________________________________________
  187. tf.__operators__.getitem_27 (Sl (None, 90)           0           input_1[0][0]                    
  188. __________________________________________________________________________________________________
  189. tf.__operators__.getitem_28 (Sl (None, 90)           0           input_1[0][0]                    
  190. __________________________________________________________________________________________________
  191. tf.__operators__.getitem_29 (Sl (None, 90)           0           input_1[0][0]                    
  192. __________________________________________________________________________________________________
  193. dense (Dense)                   (None, 90)           5850        lstm[0][0]                       
  194.                                                                  lstm[1][0]                       
  195.                                                                  lstm[2][0]                       
  196.                                                                  lstm[3][0]                       
  197.                                                                  lstm[4][0]                       
  198.                                                                  lstm[5][0]                       
  199.                                                                  lstm[6][0]                       
  200.                                                                  lstm[7][0]                       
  201.                                                                  lstm[8][0]                       
  202.                                                                  lstm[9][0]                       
  203.                                                                  lstm[10][0]                      
  204.                                                                  lstm[11][0]                      
  205.                                                                  lstm[12][0]                      
  206.                                                                  lstm[13][0]                      
  207.                                                                  lstm[14][0]                      
  208.                                                                  lstm[15][0]                      
  209.                                                                  lstm[16][0]                      
  210.                                                                  lstm[17][0]                      
  211.                                                                  lstm[18][0]                      
  212.                                                                  lstm[19][0]                      
  213.                                                                  lstm[20][0]                      
  214.                                                                  lstm[21][0]                      
  215.                                                                  lstm[22][0]                      
  216.                                                                  lstm[23][0]                      
  217.                                                                  lstm[24][0]                      
  218.                                                                  lstm[25][0]                      
  219.                                                                  lstm[26][0]                      
  220.                                                                  lstm[27][0]                      
  221.                                                                  lstm[28][0]                      
  222.                                                                  lstm[29][0]                      
  223. ==================================================================================================
  224. Total params: 45,530
  225. Trainable params: 45,530
  226. Non-trainable params: 0
  227. __________________________________________________________________________________________________
编译模型
  1. * 你现在需要编译你的模型进行训练。
  2. * 我们将使用：
  3. - optimizer：亚当优化器
  4. - 损失函数：分类交叉熵（多类分类）
  1. opt = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, decay=0.01)
  2.  
  3. model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
初始化隐藏状态和单元状态

最后，让我们初始化‘ a0 ’和‘ c0 ’，使LSTM的初始状态为零。

  1. m = 60
  2. a0 = np.zeros((m, n_a))
  3. c0 = np.zeros((m, n_a))
训练模型
history = model.fit([X, a0, c0], list(Y), epochs=100, verbose = 0)
  1. print(f"loss at epoch 1: {history.history['loss'][0]}")
  2. print(f"loss at epoch 100: {history.history['loss'][99]}")
  3. plt.plot(history.history['loss'])

loss at epoch 1: 129.90142822265625
loss at epoch 100: 9.128642082214355

预测输出

模型损失一开始会很高（100次左右），100次之后，应该是个位数。由于权重的随机初始化，这些不会是你将看到的确切数字。

现在您已经训练了一个模型，让我们进入最后一节，实现一个推理算法，并生成一些音乐！

4.3 生成音乐

你现在有了一个训练有素的模型，它已经学会了爵士独奏家的模式。你现在可以使用这个模型来合成新的音乐！

预测与采样

  1. # UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: music_inference_model
  3.  
  4. def music_inference_model(LSTM_cell, densor, Ty=100):
  5.     
  6.     # Get the shape of input values
  7.     n_values = densor.units
  8.     # Get the number of the hidden state vector
  9.     n_a = LSTM_cell.units
  10.     
  11.     # Define the input of your model with a shape 
  12.     x0 = Input(shape=(1, n_values))
  13.     
  14.     
  15.     # Define s0, initial hidden state for the decoder LSTM
  16.     a0 = Input(shape=(n_a,), name='a0')
  17.     c0 = Input(shape=(n_a,), name='c0')
  18.     a = a0
  19.     c = c0
  20.     x = x0
  21.  
  22.     # Step 1: Create an empty list of "outputs" to later store your predicted values (≈1 line)
  23.     outputs = []
  24.     
  25.     # Step 2: Loop over Ty and generate a value at every time step
  26.     for t in range(Ty):
  27.         # Step 2.A: Perform one step of LSTM_cell. Use "x", not "x0" (≈1 line)
  28.         a, _, c = LSTM_cell(x, initial_state=[a, c])
  29.         
  30.         # Step 2.B: Apply Dense layer to the hidden state output of the LSTM_cell (≈1 line)
  31.         out = densor(a)
  32.         # Step 2.C: Append the prediction "out" to "outputs". out.shape = (None, 90) (≈1 line)
  33.         outputs.append(out)
  34.  
  35.         
  36.         # Step 2.D: 
  37.         # Select the next value according to "out",
  38.         # Set "x" to be the one-hot representation of the selected value
  39.         # See instructions above.
  40.         x = tf.math.argmax(out, axis=-1)
  41.         x = tf.one_hot(x, n_values)
  42.         # Step 2.E: 
  43.         # Use RepeatVector(1) to convert x into a tensor with shape=(None, 1, 90)
  44.         x = RepeatVector(1)(x)
  45.         
  46.     # Step 3: Create model instance with the correct "inputs" and "outputs" (≈1 line)
  47.     inference_model = Model(inputs=[x0, a0, c0], outputs=outputs)
  48.     
  49.     return inference_model

该模型被硬编码为生成50个值。

inference_model = music_inference_model(LSTM_cell, densor, Ty = 50)
  1. # UNIT TEST
  2. inference_summary = summary(inference_model) 
  3. # comparator(inference_summary, music_inference_model_out)
  1. # Check the inference model
  2. inference_model.summary()
  1. Model: "model_1"
  2. __________________________________________________________________________________________________
  3. Layer (type)                    Output Shape         Param #     Connected to                     
  4. ==================================================================================================
  5. input_2 (InputLayer)            [(None, 1, 90)]      0                                            
  6. __________________________________________________________________________________________________
  7. a0 (InputLayer)                 [(None, 64)]         0                                            
  8. __________________________________________________________________________________________________
  9. c0 (InputLayer)                 [(None, 64)]         0                                            
  10. __________________________________________________________________________________________________
  11. lstm (LSTM)                     [(None, 64), (None,  39680       input_2[0][0]                    
  12.                                                                  a0[0][0]                         
  13.                                                                  c0[0][0]                         
  14.                                                                  repeat_vector[0][0]              
  15.                                                                  lstm[30][0]                      
  16.                                                                  lstm[30][2]                      
  17.                                                                  repeat_vector_1[0][0]            
  18.                                                                  lstm[31][0]                      
  19.                                                                  lstm[31][2]                      
  20.                                                                  repeat_vector_2[0][0]            
  21.                                                                  lstm[32][0]                      
  22.                                                                  lstm[32][2]                      
  23.                                                                  repeat_vector_3[0][0]            
  24.                                                                  lstm[33][0]                      
  25.                                                                  lstm[33][2]                      
  26.                                                                  repeat_vector_4[0][0]            
  27.                                                                  lstm[34][0]                      
  28.                                                                  lstm[34][2]                      
  29.                                                                  repeat_vector_5[0][0]            
  30.                                                                  lstm[35][0]                      
  31.                                                                  lstm[35][2]                      
  32.                                                                  repeat_vector_6[0][0]            
  33.                                                                  lstm[36][0]                      
  34.                                                                  lstm[36][2]                      
  35.                                                                  repeat_vector_7[0][0]            
  36.                                                                  lstm[37][0]                      
  37.                                                                  lstm[37][2]                      
  38.                                                                  repeat_vector_8[0][0]            
  39.                                                                  lstm[38][0]                      
  40.                                                                  lstm[38][2]                      
  41.                                                                  repeat_vector_9[0][0]            
  42.                                                                  lstm[39][0]                      
  43.                                                                  lstm[39][2]                      
  44.                                                                  repeat_vector_10[0][0]           
  45.                                                                  lstm[40][0]                      
  46.                                                                  lstm[40][2]                      
  47.                                                                  repeat_vector_11[0][0]           
  48.                                                                  lstm[41][0]                      
  49.                                                                  lstm[41][2]                      
  50.                                                                  repeat_vector_12[0][0]           
  51.                                                                  lstm[42][0]                      
  52.                                                                  lstm[42][2]                      
  53.                                                                  repeat_vector_13[0][0]           
  54.                                                                  lstm[43][0]                      
  55.                                                                  lstm[43][2]                      
  56.                                                                  repeat_vector_14[0][0]           
  57.                                                                  lstm[44][0]                      
  58.                                                                  lstm[44][2]                      
  59.                                                                  repeat_vector_15[0][0]           
  60.                                                                  lstm[45][0]                      
  61.                                                                  lstm[45][2]                      
  62.                                                                  repeat_vector_16[0][0]           
  63.                                                                  lstm[46][0]                      
  64.                                                                  lstm[46][2]                      
  65.                                                                  repeat_vector_17[0][0]           
  66.                                                                  lstm[47][0]                      
  67.                                                                  lstm[47][2]                      
  68.                                                                  repeat_vector_18[0][0]           
  69.                                                                  lstm[48][0]                      
  70.                                                                  lstm[48][2]                      
  71.                                                                  repeat_vector_19[0][0]           
  72.                                                                  lstm[49][0]                      
  73.                                                                  lstm[49][2]                      
  74.                                                                  repeat_vector_20[0][0]           
  75.                                                                  lstm[50][0]                      
  76.                                                                  lstm[50][2]                      
  77.                                                                  repeat_vector_21[0][0]           
  78.                                                                  lstm[51][0]                      
  79.                                                                  lstm[51][2]                      
  80.                                                                  repeat_vector_22[0][0]           
  81.                                                                  lstm[52][0]                      
  82.                                                                  lstm[52][2]                      
  83.                                                                  repeat_vector_23[0][0]           
  84.                                                                  lstm[53][0]                      
  85.                                                                  lstm[53][2]                      
  86.                                                                  repeat_vector_24[0][0]           
  87.                                                                  lstm[54][0]                      
  88.                                                                  lstm[54][2]                      
  89.                                                                  repeat_vector_25[0][0]           
  90.                                                                  lstm[55][0]                      
  91.                                                                  lstm[55][2]                      
  92.                                                                  repeat_vector_26[0][0]           
  93.                                                                  lstm[56][0]                      
  94.                                                                  lstm[56][2]                      
  95.                                                                  repeat_vector_27[0][0]           
  96.                                                                  lstm[57][0]                      
  97.                                                                  lstm[57][2]                      
  98.                                                                  repeat_vector_28[0][0]           
  99.                                                                  lstm[58][0]                      
  100.                                                                  lstm[58][2]                      
  101.                                                                  repeat_vector_29[0][0]           
  102.                                                                  lstm[59][0]                      
  103.                                                                  lstm[59][2]                      
  104.                                                                  repeat_vector_30[0][0]           
  105.                                                                  lstm[60][0]                      
  106.                                                                  lstm[60][2]                      
  107.                                                                  repeat_vector_31[0][0]           
  108.                                                                  lstm[61][0]                      
  109.                                                                  lstm[61][2]                      
  110.                                                                  repeat_vector_32[0][0]           
  111.                                                                  lstm[62][0]                      
  112.                                                                  lstm[62][2]                      
  113.                                                                  repeat_vector_33[0][0]           
  114.                                                                  lstm[63][0]                      
  115.                                                                  lstm[63][2]                      
  116.                                                                  repeat_vector_34[0][0]           
  117.                                                                  lstm[64][0]                      
  118.                                                                  lstm[64][2]                      
  119.                                                                  repeat_vector_35[0][0]           
  120.                                                                  lstm[65][0]                      
  121.                                                                  lstm[65][2]                      
  122.                                                                  repeat_vector_36[0][0]           
  123.                                                                  lstm[66][0]                      
  124.                                                                  lstm[66][2]                      
  125.                                                                  repeat_vector_37[0][0]           
  126.                                                                  lstm[67][0]                      
  127.                                                                  lstm[67][2]                      
  128.                                                                  repeat_vector_38[0][0]           
  129.                                                                  lstm[68][0]                      
  130.                                                                  lstm[68][2]                      
  131.                                                                  repeat_vector_39[0][0]           
  132.                                                                  lstm[69][0]                      
  133.                                                                  lstm[69][2]                      
  134.                                                                  repeat_vector_40[0][0]           
  135.                                                                  lstm[70][0]                      
  136.                                                                  lstm[70][2]                      
  137.                                                                  repeat_vector_41[0][0]           
  138.                                                                  lstm[71][0]                      
  139.                                                                  lstm[71][2]                      
  140.                                                                  repeat_vector_42[0][0]           
  141.                                                                  lstm[72][0]                      
  142.                                                                  lstm[72][2]                      
  143.                                                                  repeat_vector_43[0][0]           
  144.                                                                  lstm[73][0]                      
  145.                                                                  lstm[73][2]                      
  146.                                                                  repeat_vector_44[0][0]           
  147.                                                                  lstm[74][0]                      
  148.                                                                  lstm[74][2]                      
  149.                                                                  repeat_vector_45[0][0]           
  150.                                                                  lstm[75][0]                      
  151.                                                                  lstm[75][2]                      
  152.                                                                  repeat_vector_46[0][0]           
  153.                                                                  lstm[76][0]                      
  154.                                                                  lstm[76][2]                      
  155.                                                                  repeat_vector_47[0][0]           
  156.                                                                  lstm[77][0]                      
  157.                                                                  lstm[77][2]                      
  158.                                                                  repeat_vector_48[0][0]           
  159.                                                                  lstm[78][0]                      
  160.                                                                  lstm[78][2]                      
  161. __________________________________________________________________________________________________
  162. dense (Dense)                   (None, 90)           5850        lstm[30][0]                      
  163.                                                                  lstm[31][0]                      
  164.                                                                  lstm[32][0]                      
  165.                                                                  lstm[33][0]                      
  166.                                                                  lstm[34][0]                      
  167.                                                                  lstm[35][0]                      
  168.                                                                  lstm[36][0]                      
  169.                                                                  lstm[37][0]                      
  170.                                                                  lstm[38][0]                      
  171.                                                                  lstm[39][0]                      
  172.                                                                  lstm[40][0]                      
  173.                                                                  lstm[41][0]                      
  174.                                                                  lstm[42][0]                      
  175.                                                                  lstm[43][0]                      
  176.                                                                  lstm[44][0]                      
  177.                                                                  lstm[45][0]                      
  178.                                                                  lstm[46][0]                      
  179.                                                                  lstm[47][0]                      
  180.                                                                  lstm[48][0]                      
  181.                                                                  lstm[49][0]                      
  182.                                                                  lstm[50][0]                      
  183.                                                                  lstm[51][0]                      
  184.                                                                  lstm[52][0]                      
  185.                                                                  lstm[53][0]                      
  186.                                                                  lstm[54][0]                      
  187.                                                                  lstm[55][0]                      
  188.                                                                  lstm[56][0]                      
  189.                                                                  lstm[57][0]                      
  190.                                                                  lstm[58][0]                      
  191.                                                                  lstm[59][0]                      
  192.                                                                  lstm[60][0]                      
  193.                                                                  lstm[61][0]                      
  194.                                                                  lstm[62][0]                      
  195.                                                                  lstm[63][0]                      
  196.                                                                  lstm[64][0]                      
  197.                                                                  lstm[65][0]                      
  198.                                                                  lstm[66][0]                      
  199.                                                                  lstm[67][0]                      
  200.                                                                  lstm[68][0]                      
  201.                                                                  lstm[69][0]                      
  202.                                                                  lstm[70][0]                      
  203.                                                                  lstm[71][0]                      
  204.                                                                  lstm[72][0]                      
  205.                                                                  lstm[73][0]                      
  206.                                                                  lstm[74][0]                      
  207.                                                                  lstm[75][0]                      
  208.                                                                  lstm[76][0]                      
  209.                                                                  lstm[77][0]                      
  210.                                                                  lstm[78][0]                      
  211.                                                                  lstm[79][0]                      
  212. __________________________________________________________________________________________________
  213. tf.math.argmax (TFOpLambda)     (None,)              0           dense[30][0]                     
  214. __________________________________________________________________________________________________
  215. tf.one_hot (TFOpLambda)         (None, 90)           0           tf.math.argmax[0][0]             
  216. __________________________________________________________________________________________________
  217. repeat_vector (RepeatVector)    (None, 1, 90)        0           tf.one_hot[0][0]                 
  218. __________________________________________________________________________________________________
  219. tf.math.argmax_1 (TFOpLambda)   (None,)              0           dense[31][0]                     
  220. __________________________________________________________________________________________________
  221. tf.one_hot_1 (TFOpLambda)       (None, 90)           0           tf.math.argmax_1[0][0]           
  222. __________________________________________________________________________________________________
  223. repeat_vector_1 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_1[0][0]               
  224. __________________________________________________________________________________________________
  225. tf.math.argmax_2 (TFOpLambda)   (None,)              0           dense[32][0]                     
  226. __________________________________________________________________________________________________
  227. tf.one_hot_2 (TFOpLambda)       (None, 90)           0           tf.math.argmax_2[0][0]           
  228. __________________________________________________________________________________________________
  229. repeat_vector_2 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_2[0][0]               
  230. __________________________________________________________________________________________________
  231. tf.math.argmax_3 (TFOpLambda)   (None,)              0           dense[33][0]                     
  232. __________________________________________________________________________________________________
  233. tf.one_hot_3 (TFOpLambda)       (None, 90)           0           tf.math.argmax_3[0][0]           
  234. __________________________________________________________________________________________________
  235. repeat_vector_3 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_3[0][0]               
  236. __________________________________________________________________________________________________
  237. tf.math.argmax_4 (TFOpLambda)   (None,)              0           dense[34][0]                     
  238. __________________________________________________________________________________________________
  239. tf.one_hot_4 (TFOpLambda)       (None, 90)           0           tf.math.argmax_4[0][0]           
  240. __________________________________________________________________________________________________
  241. repeat_vector_4 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_4[0][0]               
  242. __________________________________________________________________________________________________
  243. tf.math.argmax_5 (TFOpLambda)   (None,)              0           dense[35][0]                     
  244. __________________________________________________________________________________________________
  245. tf.one_hot_5 (TFOpLambda)       (None, 90)           0           tf.math.argmax_5[0][0]           
  246. __________________________________________________________________________________________________
  247. repeat_vector_5 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_5[0][0]               
  248. __________________________________________________________________________________________________
  249. tf.math.argmax_6 (TFOpLambda)   (None,)              0           dense[36][0]                     
  250. __________________________________________________________________________________________________
  251. tf.one_hot_6 (TFOpLambda)       (None, 90)           0           tf.math.argmax_6[0][0]           
  252. __________________________________________________________________________________________________
  253. repeat_vector_6 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_6[0][0]               
  254. __________________________________________________________________________________________________
  255. tf.math.argmax_7 (TFOpLambda)   (None,)              0           dense[37][0]                     
  256. __________________________________________________________________________________________________
  257. tf.one_hot_7 (TFOpLambda)       (None, 90)           0           tf.math.argmax_7[0][0]           
  258. __________________________________________________________________________________________________
  259. repeat_vector_7 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_7[0][0]               
  260. __________________________________________________________________________________________________
  261. tf.math.argmax_8 (TFOpLambda)   (None,)              0           dense[38][0]                     
  262. __________________________________________________________________________________________________
  263. tf.one_hot_8 (TFOpLambda)       (None, 90)           0           tf.math.argmax_8[0][0]           
  264. __________________________________________________________________________________________________
  265. repeat_vector_8 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_8[0][0]               
  266. __________________________________________________________________________________________________
  267. tf.math.argmax_9 (TFOpLambda)   (None,)              0           dense[39][0]                     
  268. __________________________________________________________________________________________________
  269. tf.one_hot_9 (TFOpLambda)       (None, 90)           0           tf.math.argmax_9[0][0]           
  270. __________________________________________________________________________________________________
  271. repeat_vector_9 (RepeatVector)  (None, 1, 90)        0           tf.one_hot_9[0][0]               
  272. __________________________________________________________________________________________________
  273. tf.math.argmax_10 (TFOpLambda)  (None,)              0           dense[40][0]                     
  274. __________________________________________________________________________________________________
  275. tf.one_hot_10 (TFOpLambda)      (None, 90)           0           tf.math.argmax_10[0][0]          
  276. __________________________________________________________________________________________________
  277. repeat_vector_10 (RepeatVector) (None, 1, 90)        0           tf.one_hot_10[0][0]              
  278. __________________________________________________________________________________________________
  279. tf.math.argmax_11 (TFOpLambda)  (None,)              0           dense[41][0]                     
  280. __________________________________________________________________________________________________
  281. tf.one_hot_11 (TFOpLambda)      (None, 90)           0           tf.math.argmax_11[0][0]          
  282. __________________________________________________________________________________________________
  283. repeat_vector_11 (RepeatVector) (None, 1, 90)        0           tf.one_hot_11[0][0]              
  284. __________________________________________________________________________________________________
  285. tf.math.argmax_12 (TFOpLambda)  (None,)              0           dense[42][0]                     
  286. __________________________________________________________________________________________________
  287. tf.one_hot_12 (TFOpLambda)      (None, 90)           0           tf.math.argmax_12[0][0]          
  288. __________________________________________________________________________________________________
  289. repeat_vector_12 (RepeatVector) (None, 1, 90)        0           tf.one_hot_12[0][0]              
  290. __________________________________________________________________________________________________
  291. tf.math.argmax_13 (TFOpLambda)  (None,)              0           dense[43][0]                     
  292. __________________________________________________________________________________________________
  293. tf.one_hot_13 (TFOpLambda)      (None, 90)           0           tf.math.argmax_13[0][0]          
  294. __________________________________________________________________________________________________
  295. repeat_vector_13 (RepeatVector) (None, 1, 90)        0           tf.one_hot_13[0][0]              
  296. __________________________________________________________________________________________________
  297. tf.math.argmax_14 (TFOpLambda)  (None,)              0           dense[44][0]                     
  298. __________________________________________________________________________________________________
  299. tf.one_hot_14 (TFOpLambda)      (None, 90)           0           tf.math.argmax_14[0][0]          
  300. __________________________________________________________________________________________________
  301. repeat_vector_14 (RepeatVector) (None, 1, 90)        0           tf.one_hot_14[0][0]              
  302. __________________________________________________________________________________________________
  303. tf.math.argmax_15 (TFOpLambda)  (None,)              0           dense[45][0]                     
  304. __________________________________________________________________________________________________
  305. tf.one_hot_15 (TFOpLambda)      (None, 90)           0           tf.math.argmax_15[0][0]          
  306. __________________________________________________________________________________________________
  307. repeat_vector_15 (RepeatVector) (None, 1, 90)        0           tf.one_hot_15[0][0]              
  308. __________________________________________________________________________________________________
  309. tf.math.argmax_16 (TFOpLambda)  (None,)              0           dense[46][0]                     
  310. __________________________________________________________________________________________________
  311. tf.one_hot_16 (TFOpLambda)      (None, 90)           0           tf.math.argmax_16[0][0]          
  312. __________________________________________________________________________________________________
  313. repeat_vector_16 (RepeatVector) (None, 1, 90)        0           tf.one_hot_16[0][0]              
  314. __________________________________________________________________________________________________
  315. tf.math.argmax_17 (TFOpLambda)  (None,)              0           dense[47][0]                     
  316. __________________________________________________________________________________________________
  317. tf.one_hot_17 (TFOpLambda)      (None, 90)           0           tf.math.argmax_17[0][0]          
  318. __________________________________________________________________________________________________
  319. repeat_vector_17 (RepeatVector) (None, 1, 90)        0           tf.one_hot_17[0][0]              
  320. __________________________________________________________________________________________________
  321. tf.math.argmax_18 (TFOpLambda)  (None,)              0           dense[48][0]                     
  322. __________________________________________________________________________________________________
  323. tf.one_hot_18 (TFOpLambda)      (None, 90)           0           tf.math.argmax_18[0][0]          
  324. __________________________________________________________________________________________________
  325. repeat_vector_18 (RepeatVector) (None, 1, 90)        0           tf.one_hot_18[0][0]              
  326. __________________________________________________________________________________________________
  327. tf.math.argmax_19 (TFOpLambda)  (None,)              0           dense[49][0]                     
  328. __________________________________________________________________________________________________
  329. tf.one_hot_19 (TFOpLambda)      (None, 90)           0           tf.math.argmax_19[0][0]          
  330. __________________________________________________________________________________________________
  331. repeat_vector_19 (RepeatVector) (None, 1, 90)        0           tf.one_hot_19[0][0]              
  332. __________________________________________________________________________________________________
  333. tf.math.argmax_20 (TFOpLambda)  (None,)              0           dense[50][0]                     
  334. __________________________________________________________________________________________________
  335. tf.one_hot_20 (TFOpLambda)      (None, 90)           0           tf.math.argmax_20[0][0]          
  336. __________________________________________________________________________________________________
  337. repeat_vector_20 (RepeatVector) (None, 1, 90)        0           tf.one_hot_20[0][0]              
  338. __________________________________________________________________________________________________
  339. tf.math.argmax_21 (TFOpLambda)  (None,)              0           dense[51][0]                     
  340. __________________________________________________________________________________________________
  341. tf.one_hot_21 (TFOpLambda)      (None, 90)           0           tf.math.argmax_21[0][0]          
  342. __________________________________________________________________________________________________
  343. repeat_vector_21 (RepeatVector) (None, 1, 90)        0           tf.one_hot_21[0][0]              
  344. __________________________________________________________________________________________________
  345. tf.math.argmax_22 (TFOpLambda)  (None,)              0           dense[52][0]                     
  346. __________________________________________________________________________________________________
  347. tf.one_hot_22 (TFOpLambda)      (None, 90)           0           tf.math.argmax_22[0][0]          
  348. __________________________________________________________________________________________________
  349. repeat_vector_22 (RepeatVector) (None, 1, 90)        0           tf.one_hot_22[0][0]              
  350. __________________________________________________________________________________________________
  351. tf.math.argmax_23 (TFOpLambda)  (None,)              0           dense[53][0]                     
  352. __________________________________________________________________________________________________
  353. tf.one_hot_23 (TFOpLambda)      (None, 90)           0           tf.math.argmax_23[0][0]          
  354. __________________________________________________________________________________________________
  355. repeat_vector_23 (RepeatVector) (None, 1, 90)        0           tf.one_hot_23[0][0]              
  356. __________________________________________________________________________________________________
  357. tf.math.argmax_24 (TFOpLambda)  (None,)              0           dense[54][0]                     
  358. __________________________________________________________________________________________________
  359. tf.one_hot_24 (TFOpLambda)      (None, 90)           0           tf.math.argmax_24[0][0]          
  360. __________________________________________________________________________________________________
  361. repeat_vector_24 (RepeatVector) (None, 1, 90)        0           tf.one_hot_24[0][0]              
  362. __________________________________________________________________________________________________
  363. tf.math.argmax_25 (TFOpLambda)  (None,)              0           dense[55][0]                     
  364. __________________________________________________________________________________________________
  365. tf.one_hot_25 (TFOpLambda)      (None, 90)           0           tf.math.argmax_25[0][0]          
  366. __________________________________________________________________________________________________
  367. repeat_vector_25 (RepeatVector) (None, 1, 90)        0           tf.one_hot_25[0][0]              
  368. __________________________________________________________________________________________________
  369. tf.math.argmax_26 (TFOpLambda)  (None,)              0           dense[56][0]                     
  370. __________________________________________________________________________________________________
  371. tf.one_hot_26 (TFOpLambda)      (None, 90)           0           tf.math.argmax_26[0][0]          
  372. __________________________________________________________________________________________________
  373. repeat_vector_26 (RepeatVector) (None, 1, 90)        0           tf.one_hot_26[0][0]              
  374. __________________________________________________________________________________________________
  375. tf.math.argmax_27 (TFOpLambda)  (None,)              0           dense[57][0]                     
  376. __________________________________________________________________________________________________
  377. tf.one_hot_27 (TFOpLambda)      (None, 90)           0           tf.math.argmax_27[0][0]          
  378. __________________________________________________________________________________________________
  379. repeat_vector_27 (RepeatVector) (None, 1, 90)        0           tf.one_hot_27[0][0]              
  380. __________________________________________________________________________________________________
  381. tf.math.argmax_28 (TFOpLambda)  (None,)              0           dense[58][0]                     
  382. __________________________________________________________________________________________________
  383. tf.one_hot_28 (TFOpLambda)      (None, 90)           0           tf.math.argmax_28[0][0]          
  384. __________________________________________________________________________________________________
  385. repeat_vector_28 (RepeatVector) (None, 1, 90)        0           tf.one_hot_28[0][0]              
  386. __________________________________________________________________________________________________
  387. tf.math.argmax_29 (TFOpLambda)  (None,)              0           dense[59][0]                     
  388. __________________________________________________________________________________________________
  389. tf.one_hot_29 (TFOpLambda)      (None, 90)           0           tf.math.argmax_29[0][0]          
  390. __________________________________________________________________________________________________
  391. repeat_vector_29 (RepeatVector) (None, 1, 90)        0           tf.one_hot_29[0][0]              
  392. __________________________________________________________________________________________________
  393. tf.math.argmax_30 (TFOpLambda)  (None,)              0           dense[60][0]                     
  394. __________________________________________________________________________________________________
  395. tf.one_hot_30 (TFOpLambda)      (None, 90)           0           tf.math.argmax_30[0][0]          
  396. __________________________________________________________________________________________________
  397. repeat_vector_30 (RepeatVector) (None, 1, 90)        0           tf.one_hot_30[0][0]              
  398. __________________________________________________________________________________________________
  399. tf.math.argmax_31 (TFOpLambda)  (None,)              0           dense[61][0]                     
  400. __________________________________________________________________________________________________
  401. tf.one_hot_31 (TFOpLambda)      (None, 90)           0           tf.math.argmax_31[0][0]          
  402. __________________________________________________________________________________________________
  403. repeat_vector_31 (RepeatVector) (None, 1, 90)        0           tf.one_hot_31[0][0]              
  404. __________________________________________________________________________________________________
  405. tf.math.argmax_32 (TFOpLambda)  (None,)              0           dense[62][0]                     
  406. __________________________________________________________________________________________________
  407. tf.one_hot_32 (TFOpLambda)      (None, 90)           0           tf.math.argmax_32[0][0]          
  408. __________________________________________________________________________________________________
  409. repeat_vector_32 (RepeatVector) (None, 1, 90)        0           tf.one_hot_32[0][0]              
  410. __________________________________________________________________________________________________
  411. tf.math.argmax_33 (TFOpLambda)  (None,)              0           dense[63][0]                     
  412. __________________________________________________________________________________________________
  413. tf.one_hot_33 (TFOpLambda)      (None, 90)           0           tf.math.argmax_33[0][0]          
  414. __________________________________________________________________________________________________
  415. repeat_vector_33 (RepeatVector) (None, 1, 90)        0           tf.one_hot_33[0][0]              
  416. __________________________________________________________________________________________________
  417. tf.math.argmax_34 (TFOpLambda)  (None,)              0           dense[64][0]                     
  418. __________________________________________________________________________________________________
  419. tf.one_hot_34 (TFOpLambda)      (None, 90)           0           tf.math.argmax_34[0][0]          
  420. __________________________________________________________________________________________________
  421. repeat_vector_34 (RepeatVector) (None, 1, 90)        0           tf.one_hot_34[0][0]              
  422. __________________________________________________________________________________________________
  423. tf.math.argmax_35 (TFOpLambda)  (None,)              0           dense[65][0]                     
  424. __________________________________________________________________________________________________
  425. tf.one_hot_35 (TFOpLambda)      (None, 90)           0           tf.math.argmax_35[0][0]          
  426. __________________________________________________________________________________________________
  427. repeat_vector_35 (RepeatVector) (None, 1, 90)        0           tf.one_hot_35[0][0]              
  428. __________________________________________________________________________________________________
  429. tf.math.argmax_36 (TFOpLambda)  (None,)              0           dense[66][0]                     
  430. __________________________________________________________________________________________________
  431. tf.one_hot_36 (TFOpLambda)      (None, 90)           0           tf.math.argmax_36[0][0]          
  432. __________________________________________________________________________________________________
  433. repeat_vector_36 (RepeatVector) (None, 1, 90)        0           tf.one_hot_36[0][0]              
  434. __________________________________________________________________________________________________
  435. tf.math.argmax_37 (TFOpLambda)  (None,)              0           dense[67][0]                     
  436. __________________________________________________________________________________________________
  437. tf.one_hot_37 (TFOpLambda)      (None, 90)           0           tf.math.argmax_37[0][0]          
  438. __________________________________________________________________________________________________
  439. repeat_vector_37 (RepeatVector) (None, 1, 90)        0           tf.one_hot_37[0][0]              
  440. __________________________________________________________________________________________________
  441. tf.math.argmax_38 (TFOpLambda)  (None,)              0           dense[68][0]                     
  442. __________________________________________________________________________________________________
  443. tf.one_hot_38 (TFOpLambda)      (None, 90)           0           tf.math.argmax_38[0][0]          
  444. __________________________________________________________________________________________________
  445. repeat_vector_38 (RepeatVector) (None, 1, 90)        0           tf.one_hot_38[0][0]              
  446. __________________________________________________________________________________________________
  447. tf.math.argmax_39 (TFOpLambda)  (None,)              0           dense[69][0]                     
  448. __________________________________________________________________________________________________
  449. tf.one_hot_39 (TFOpLambda)      (None, 90)           0           tf.math.argmax_39[0][0]          
  450. __________________________________________________________________________________________________
  451. repeat_vector_39 (RepeatVector) (None, 1, 90)        0           tf.one_hot_39[0][0]              
  452. __________________________________________________________________________________________________
  453. tf.math.argmax_40 (TFOpLambda)  (None,)              0           dense[70][0]                     
  454. __________________________________________________________________________________________________
  455. tf.one_hot_40 (TFOpLambda)      (None, 90)           0           tf.math.argmax_40[0][0]          
  456. __________________________________________________________________________________________________
  457. repeat_vector_40 (RepeatVector) (None, 1, 90)        0           tf.one_hot_40[0][0]              
  458. __________________________________________________________________________________________________
  459. tf.math.argmax_41 (TFOpLambda)  (None,)              0           dense[71][0]                     
  460. __________________________________________________________________________________________________
  461. tf.one_hot_41 (TFOpLambda)      (None, 90)           0           tf.math.argmax_41[0][0]          
  462. __________________________________________________________________________________________________
  463. repeat_vector_41 (RepeatVector) (None, 1, 90)        0           tf.one_hot_41[0][0]              
  464. __________________________________________________________________________________________________
  465. tf.math.argmax_42 (TFOpLambda)  (None,)              0           dense[72][0]                     
  466. __________________________________________________________________________________________________
  467. tf.one_hot_42 (TFOpLambda)      (None, 90)           0           tf.math.argmax_42[0][0]          
  468. __________________________________________________________________________________________________
  469. repeat_vector_42 (RepeatVector) (None, 1, 90)        0           tf.one_hot_42[0][0]              
  470. __________________________________________________________________________________________________
  471. tf.math.argmax_43 (TFOpLambda)  (None,)              0           dense[73][0]                     
  472. __________________________________________________________________________________________________
  473. tf.one_hot_43 (TFOpLambda)      (None, 90)           0           tf.math.argmax_43[0][0]          
  474. __________________________________________________________________________________________________
  475. repeat_vector_43 (RepeatVector) (None, 1, 90)        0           tf.one_hot_43[0][0]              
  476. __________________________________________________________________________________________________
  477. tf.math.argmax_44 (TFOpLambda)  (None,)              0           dense[74][0]                     
  478. __________________________________________________________________________________________________
  479. tf.one_hot_44 (TFOpLambda)      (None, 90)           0           tf.math.argmax_44[0][0]          
  480. __________________________________________________________________________________________________
  481. repeat_vector_44 (RepeatVector) (None, 1, 90)        0           tf.one_hot_44[0][0]              
  482. __________________________________________________________________________________________________
  483. tf.math.argmax_45 (TFOpLambda)  (None,)              0           dense[75][0]                     
  484. __________________________________________________________________________________________________
  485. tf.one_hot_45 (TFOpLambda)      (None, 90)           0           tf.math.argmax_45[0][0]          
  486. __________________________________________________________________________________________________
  487. repeat_vector_45 (RepeatVector) (None, 1, 90)        0           tf.one_hot_45[0][0]              
  488. __________________________________________________________________________________________________
  489. tf.math.argmax_46 (TFOpLambda)  (None,)              0           dense[76][0]                     
  490. __________________________________________________________________________________________________
  491. tf.one_hot_46 (TFOpLambda)      (None, 90)           0           tf.math.argmax_46[0][0]          
  492. __________________________________________________________________________________________________
  493. repeat_vector_46 (RepeatVector) (None, 1, 90)        0           tf.one_hot_46[0][0]              
  494. __________________________________________________________________________________________________
  495. tf.math.argmax_47 (TFOpLambda)  (None,)              0           dense[77][0]                     
  496. __________________________________________________________________________________________________
  497. tf.one_hot_47 (TFOpLambda)      (None, 90)           0           tf.math.argmax_47[0][0]          
  498. __________________________________________________________________________________________________
  499. repeat_vector_47 (RepeatVector) (None, 1, 90)        0           tf.one_hot_47[0][0]              
  500. __________________________________________________________________________________________________
  501. tf.math.argmax_48 (TFOpLambda)  (None,)              0           dense[78][0]                     
  502. __________________________________________________________________________________________________
  503. tf.one_hot_48 (TFOpLambda)      (None, 90)           0           tf.math.argmax_48[0][0]          
  504. __________________________________________________________________________________________________
  505. repeat_vector_48 (RepeatVector) (None, 1, 90)        0           tf.one_hot_48[0][0]              
  506. ==================================================================================================
  507. Total params: 45,530
  508. Trainable params: 45,530
  509. Non-trainable params: 0
  510. __________________________________________________________________________________________________

下面的代码创建了将用于初始化‘ x ’和LSTM状态变量‘ a ’和‘ c ’的零值向量。

  1. x_initializer = np.zeros((1, 1, n_values))
  2. a_initializer = np.zeros((1, n_a))
  3. c_initializer = np.zeros((1, n_a))
  1. # UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
  2. # GRADED FUNCTION: predict_and_sample
  3.  
  4. def predict_and_sample(inference_model, x_initializer = x_initializer, a_initializer = a_initializer, 
  5.                        c_initializer = c_initializer):
  6.     
  7.     n_values = x_initializer.shape[2]
  8.     
  9.     # Step 1: Use your inference model to predict an output sequence given x_initializer, a_initializer and c_initializer.
  10.     pred = inference_model.predict([x_initializer, a_initializer, c_initializer])
  11.     # Step 2: Convert "pred" into an np.array() of indices with the maximum probabilities
  12.     indices = np.argmax(pred, axis = -1)
  13.     # Step 3: Convert indices to one-hot vectors, the shape of the results should be (Ty, n_values)
  14.     results = to_categorical(indices, num_classes=n_values)
  15.     
  16.     return results, indices
  1. results, indices = predict_and_sample(inference_model, x_initializer, a_initializer, c_initializer)
  2.  
  3. print("np.argmax(results[12]) =", np.argmax(results[12]))
  4. print("np.argmax(results[17]) =", np.argmax(results[17]))
  5. print("list(indices[12:18]) =", list(indices[12:18]))

np.argmax(results[12]) = 31
np.argmax(results[17]) = 44
list(indices[12:18]) = [array([31], dtype=int64), array([76], dtype=int64), array([50], dtype=int64), array([9], dtype=int64), array([34], dtype=int64), array([44], dtype=int64)] 

生成音乐

你的RNN生成一个值序列。下面的代码通过首先调用‘ predict_and_sample() ’函数来生成音乐。然后将这些值后处理成和弦（意味着可以同时播放多个值或音符）。

大多数计算音乐算法使用一些后处理，因为没有后处理就很难生成好听的音乐。后期处理的工作包括清理生成的音频，确保相同的声音不会重复太多次，或者两个连续的音符在音高上不会相差太远，等等。

有人可能会说，许多这些后期处理步骤是hack；此外，许多音乐生成文献也专注于手工制作后处理器，许多输出质量取决于后处理的质量，而不仅仅是模型的质量。但是这种后处理确实会产生巨大的差异，因此您也应该在实现中使用它。

让我们做点音乐吧！

运行以下单元格生成音乐并将其记录到您的‘ out_stream ’中。这可能需要几分钟。

out_stream = generate_music(inference_model, indices_values, chords, "output/sghn.midi")

Predicting new values for different set of chords.
Generated 37 sounds using the predicted values for the set of chords ("1") and after pruning
Generated 37 sounds using the predicted values for the set of chords ("2") and after pruning
Generated 37 sounds using the predicted values for the set of chords ("3") and after pruning
Generated 37 sounds using the predicted values for the set of chords ("4") and after pruning
Generated 37 sounds using the predicted values for the set of chords ("5") and after pruning
Your generated music is saved in output/sghn.midi 

使用基本的midi到wav解析器，您可以对该模型生成的音频剪辑有一个大致的了解。解析器非常有限。

  1. mid2wav('output/sghn.midi')
  2. IPython.display.Audio('./output/rendered.wav')

将 MIDI 文件转换为 WAV 音频文件，转换后的 WAV 文件通常保存为 rendered.wav 或指定的文件名

我觉得还蛮好听的，可惜分享不了。 

恭喜你!

你已经完成了这个任务，并生成了你自己的爵士独奏！科尔特兰一家会感到骄傲的。

到目前为止，你已经：

- 将LSTM应用于音乐生成任务

- 通过深度学习生成自己的爵士音乐

- 使用灵活的Functional API创建更复杂的模型

这是一项漫长的任务。你应该为你的努力感到自豪，希望你能有一些好音乐来展示你的努力。谢谢，下次见！

注意事项：

  1. - 序列模型可以用来生成音乐值，然后后期处理成midi音乐。
  2. - 你可以使用一个相当相似的模型来完成从生成恐龙名字到生成原创音乐的任务，唯一的主要区别是输入到模型中。
  3. - 在Keras中，序列生成涉及定义具有共享权重的层，然后在不同的时间步长$1,\ldots, T_x$中重复。
注：本文转载自blog.csdn.net的笨笨sg的文章"https://blog.csdn.net/a131529/article/details/143836250"。版权归原作者所有，此博客不拥有其著作权，亦不承担相应法律责任。如有侵权，请联系我们删除。
复制链接
相关推荐
发表评论

评论记录：

未查询到任何数据！