Keras LSTM - 分类交叉熵降至 0

Question

我目前正在尝试比较一些 RNN，但我只有 LSTM 有问题，我不知道为什么。

我正在使用相同的 code/dataset LSTM、SimpleRNN 和 GRU 进行训练。对于所有这些，损失正常减少。但是对于 LSTM，在某个点之后（损失在 0.4 左右），损失直接下降到 10e-8。如果我尝试预测输出，我只有 Nan。

这是代码：

nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64

model = Sequential()

model.add(LSTM( units=nb_unit, 
                input_shape=inp_shape, 
                return_sequences=True, 
                activation='softmax'))  # I just change the cell name
model.compile(loss=loss_,
              optimizer=optimizer_,
              metrics=[metrics_])

checkpoint = ModelCheckpoint("lstm_simple.h5",
                            monitor=loss_,
                            verbose=1,
                            save_best_only=True,
                            save_weights_only=False,
                            mode='auto',
                            period=1)
early = EarlyStopping( monitor='loss',
                       min_delta=0,
                       patience=10,
                       verbose=1,
                       mode='auto')

history = model.fit(X_train, y_train, 
                    validation_data=(X_test, y_test), 
                    epochs=nb_epoch, 
                    batch_size=batch_size, 
                    verbose=2, 
                    callbacks = [checkpoint, early])

这是具有相同输入的 GRU 和 LSTM 的输出：

Input :
[[[1 0 0 0 0 0 0]
  [0 1 0 0 0 0 0]
  [0 0 0 1 0 0 0]
  [0 0 0 1 0 0 0]
  [0 1 0 0 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 1 0 0]
  [0 0 0 1 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 1 0 0]
  [0 0 0 1 0 0 0]
  [0 1 0 0 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 1 0 0]
  [0 0 0 1 0 0 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 0 1 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]]


LSTM predicts :
[[[ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]
  [ nan  nan  nan  nan  nan  nan  nan]]]


GRU predicts :
[[[ 0.     0.54   0.     0.     0.407  0.     0.   ]
  [ 0.     0.005  0.66   0.314  0.     0.     0.001]
  [ 0.     0.001  0.032  0.957  0.     0.004  0.   ]
  [ 0.     0.628  0.     0.     0.     0.372  0.   ]
  [ 0.     0.555  0.     0.     0.     0.372  0.   ]
  [ 0.     0.     0.     0.     0.996  0.319  0.   ]
  [ 0.     0.     0.167  0.55   0.     0.     0.   ]
  [ 0.     0.486  0.     0.002  0.     0.51   0.   ]
  [ 0.     0.001  0.     0.     0.992  0.499  0.   ]
  [ 0.     0.     0.301  0.55   0.     0.     0.   ]
  [ 0.     0.396  0.001  0.007  0.     0.592  0.   ]
  [ 0.     0.689  0.     0.     0.     0.592  0.   ]
  [ 0.     0.001  0.     0.     0.997  0.592  0.   ]
  [ 0.     0.     0.37   0.55   0.     0.     0.   ]
  [ 0.     0.327  0.003  0.025  0.     0.599  0.   ]
  [ 0.     0.001  0.     0.     0.967  0.599  0.002]
  [ 0.     0.     0.     0.     0.     0.002  0.874]
  [ 0.004  0.076  0.128  0.337  0.02   0.069  0.378]
  [ 0.006  0.379  0.047  0.113  0.029  0.284  0.193]
  [ 0.006  0.469  0.001  0.037  0.13   0.295  0.193]]]

对于损失，您可以在 fit() 历史的最后几行下面找到：

Epoch 116/250
Epoch 00116: categorical_crossentropy did not improve
 - 2s - loss: 0.3774 - categorical_crossentropy: 0.3774 - val_loss: 0.3945 - val_categorical_crossentropy: 0.3945

Epoch 117/250
Epoch 00117: categorical_crossentropy improved from 0.37673 to 0.08198, saving model to lstm_simple.h5
 - 2s - loss: 0.0820 - categorical_crossentropy: 0.0820 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08

Epoch 118/250
Epoch 00118: categorical_crossentropy improved from 0.08198 to 0.00000, saving model to lstm_simple.h5
 - 2s - loss: 7.5460e-08 - categorical_crossentropy: 7.5460e-08 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08

或者损失基于Epochs的演化。

我之前在没有 Softmax 和 MSE 作为损失函数的情况下尝试过，我没有得到任何错误。

如果需要，您可以在 Github (https://github.com/Coni63/SO/blob/master/Reber.ipynb) 上找到用于生成数据集的笔记本和脚本。

非常感谢您的支持，问候，尼古拉斯

编辑 1：

根本原因似乎是 Softmax 函数消失了。如果我在它崩溃之前停止它并显示我拥有的每个时间步长的 softmax 总和：

LSTM :
[[ 0.112]
 [ 0.008]
 [ 0.379]
 [ 0.04 ]
 [ 0.001]
 [ 0.104]
 [ 0.021]
 [ 0.   ]
 [ 0.104]
 [ 0.343]
 [ 0.012]
 [ 0.   ]
 [ 0.23 ]
 [ 0.13 ]
 [ 0.147]
 [ 0.145]
 [ 0.152]
 [ 0.157]
 [ 0.163]
 [ 0.169]]


GRU :
[[ 0.974]
 [ 0.807]
 [ 0.719]
 [ 1.184]
 [ 0.944]
 [ 0.999]
 [ 1.426]
 [ 0.957]
 [ 0.999]
 [ 1.212]
 [ 1.52 ]
 [ 0.954]
 [ 0.42 ]
 [ 0.83 ]
 [ 0.903]
 [ 0.944]
 [ 0.976]
 [ 1.005]
 [ 1.022]
 [ 1.029]]

Softmax 为 0，下一步将尝试除以 0。现在我不知道如何解决它。

Answer 1

我只是post我目前的解决方案，以防其他人将来遇到这个问题。

为了避免消失，我添加了一个简单的全连接层，其输出大小与输入相同，之后它就可以正常工作了。该层允许 LSTM/GRU/SRNN 输出的另一个 "configuration" 并避免输出消失。

这是最终代码：

nb_unit = 7
inp_shape = (maxlen, 7)
loss_ = "categorical_crossentropy"
metrics_ = "categorical_crossentropy"
optimizer_ = "Nadam"
nb_epoch = 250
batch_size = 64

model = Sequential()

model.add(LSTM(units=nb_unit, 
               input_shape=inp_shape, 
               return_sequences=True))     # LSTG/GRU/SimpleRNN
model.add(Dense(7, activation='softmax'))  # New
model.compile(loss=loss_,
              optimizer=optimizer_,
              metrics=[metrics_])

checkpoint = ModelCheckpoint("lstm_simple.h5",
    monitor=loss_,
    verbose=1,
    save_best_only=True,
    save_weights_only=False,
    mode='auto',
    period=1)
early = EarlyStopping(
    monitor='loss',
    min_delta=0,
    patience=10,
    verbose=1,
    mode='auto')

我希望这可以帮助其他人:)

Keras LSTM - 分类交叉熵降至 0

Keras LSTM - Categorical Cross Entropy falls to 0

lstm

keras

recurrent-neural-network

编辑 1：