CNN 中超参数的优化
Optimization of Hyperparameters in a CNN
编辑：我按照建议调整了模型。这意味着我将 lr 和 dropout 作为参数包含在 ConvNet 函数中。
我是神经网络和 CNN 的新手，面临着有关超参数优化 的问题。所以现在我将解释我到目前为止的过程：
在各种优秀 Blog-Posts 的帮助下，我能够构建适用于我的项目的 CNN。在我的项目中，我试图借助 FOMC 会议声明来预测 VIX 和标准普尔 500 指数。所以基本上我一方面使用文本数据，另一方面使用财务数据 (returns)。在预处理和应用 Googles Word2Vec pre-trained Word-Embeddings 之后，我构建了以下卷积网络：
def ConvNet(embeddings, max_sequence_length, num_words, embedding_dim, trainable=False, extra_conv=True,
lr=0.001, dropout=0.5):
embedding_layer = Embedding(num_words,
embedding_dim,
weights=[embeddings],
input_length=max_sequence_length,
trainable=trainable)
sequence_input = Input(shape=(max_sequence_length,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)
convs = []
filter_sizes = [3, 4, 5]
for filter_size in filter_sizes:
l_conv = Conv1D(filters=128, kernel_size=filter_size, activation='relu')(embedded_sequences)
l_pool = MaxPooling1D(pool_size=3)(l_conv)
convs.append(l_pool)
l_merge = concatenate([convs[0], convs[1], convs[2]], axis=1)
# add a 1D convnet with global maxpooling, instead of Yoon Kim model
conv = Conv1D(filters=128, kernel_size=3, activation='relu')(embedded_sequences)
pool = MaxPooling1D(pool_size=3)(conv)
if extra_conv == True:
x = Dropout(dropout)(l_merge)
else:
# Original Yoon Kim model
x = Dropout(dropout)(pool)
x = Flatten()(x)
x = Dense(128, activation='relu')(x)
preds = Dense(1, activation='linear')(x)
model = Model(sequence_input, preds)
sgd = SGD(learning_rate = lr, momentum= 0.8)
model.compile(loss='mean_squared_error',
optimizer= sgd,
metrics=['mean_squared_error'])
model.summary()
return model
我的模型架构是这样的：
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, 1086) 0
__________________________________________________________________________________________________
embedding_1 (Embedding) (None, 1086, 300) 532500 input_1[0][0]
__________________________________________________________________________________________________
conv1d_1 (Conv1D) (None, 1084, 128) 115328 embedding_1[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D) (None, 1083, 128) 153728 embedding_1[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D) (None, 1082, 128) 192128 embedding_1[0][0]
__________________________________________________________________________________________________
max_pooling1d_1 (MaxPooling1D) (None, 361, 128) 0 conv1d_1[0][0]
__________________________________________________________________________________________________
max_pooling1d_2 (MaxPooling1D) (None, 361, 128) 0 conv1d_2[0][0]
__________________________________________________________________________________________________
max_pooling1d_3 (MaxPooling1D) (None, 360, 128) 0 conv1d_3[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 1082, 128) 0 max_pooling1d_1[0][0]
max_pooling1d_2[0][0]
max_pooling1d_3[0][0]
__________________________________________________________________________________________________
dropout_2 (Dropout) (None, 1082, 128) 0 concatenate_1[0][0]
__________________________________________________________________________________________________
flatten_1 (Flatten) (None, 138496) 0 dropout_2[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 128) 17727616 flatten_1[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 1) 129 dense_3[0][0]
==================================================================================================
Total params: 18,721,429
Trainable params: 18,188,929
Non-trainable params: 532,500
所以，现在我面临下一个大问题，我真的运行不知道如何解决：超参数优化
我的问题是，到目前为止我发现的每个代码示例都应用于超参数的优化是架构：
model = Sequential()
embedding = model.add(layers.Embedding(MAX_VOCAB_SIZE, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH))
model.add(layers.Conv1D(filters=128, kernel_size=5, activation='relu'))
model.add(layers.MaxPool1D(pool_size=3))
model.add(Dropout(0.25))
所以我的具体问题是，如何执行超参数优化，因为每当我更改我的 ConvNet 中的某些内容时，我都会收到错误，正如我所说，我能找到的所有教程都应用于 model = Sequential()。
新的错误信息是：
__________________________________________________________________________________________________
0%| | 0/100 [00:00<?, ?trial/s, best loss=?]
job exception: 'Model' object is not subscriptable
Traceback (most recent call last):
File "/Users/lukaskoston/Desktop/MasterarbeitFOMCAnalysis/07_Regression/CNN regression neu.py", line 262, in <module>
max_evals=100)
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 482, in fmin
show_progressbar=show_progressbar,
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/base.py", line 686, in fmin
show_progressbar=show_progressbar,
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 509, in fmin
rval.exhaust()
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 330, in exhaust
self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 286, in run
self.serial_evaluate()
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 165, in serial_evaluate
result = self.domain.evaluate(spec, ctrl)
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/base.py", line 894, in evaluate
rval = self.fn(pyll_rval)
File "/Users/lukaskoston/Desktop/MasterarbeitFOMCAnalysis/07_Regression/CNN regression neu.py", line 248, in train_and_score
return hist['val_loss'][-1]
TypeError: 'Model' object is not subscriptable
提前致谢，
卢卡斯
您应该为您的方法设置超参数参数。
def ConvNet(embeddings, max_sequence_length, num_words, embedding_dim, trainable=False, extra_conv=True
lr=1.0, dropout=0.5):
# ...
然后您可以更新您的代码以使用这些值而不是固定值或 Keras 默认值。
if extra_conv == True:
x = Dropout(dropout)(l_merge)
else:
# Original Yoon Kim model
x = Dropout(dropout)(pool)
并且：
model.compile(loss='mean_squared_error',
optimizer=keras.optimizers.Adadelta(learning_rate=lr).
metrics=['mean_squared_error'])
我将从这两个开始，然后将批量大小和纪元计数留到以后。这些会对 运行 时间产生很大影响，这在超参数优化中很难解释。
然后您可以使用 hyperopt.
这样的库进行优化
from hyperopt import fmin, hp, tpe, space_eval
def train_and_score(args):
# Train the model the fixed params plus the optimization args.
# Note that this method should return the final History object.
hist = ConvNet(embeddings, max_sequence_length, num_words,
embedding_dim, trainable=False, extra_conv=True,
lr=args['lr'], dropout=args['dropout'])
# Unpack and return the last validation loss from the history.
return hist['val_loss'][-1]
# Define the space to optimize over.
space = {
'lr': hp.loguniform('lr', np.log(0.1), np.log(10.0)),
'dropout': hp.uniform('dropout', 0, 1),
}
# Minimize the training score over the space.
trials = Trials()
best = fmin(train_and_score, space, trials=trials, algo=tpe.suggest, max_evals=100)
# Print details about the best results and hyperparameters.
print(best)
print(space_eval(space, best))
还有一些库可以帮助您直接将其与 Keras 集成。一个流行的选择是 hyperas。在那种情况下，您将修改您的函数以使用一些模板而不是参数，但它在其他方面非常相似。
编辑：我按照建议调整了模型。这意味着我将 lr 和 dropout 作为参数包含在 ConvNet 函数中。
我是神经网络和 CNN 的新手，面临着有关超参数优化 的问题。所以现在我将解释我到目前为止的过程： 在各种优秀 Blog-Posts 的帮助下，我能够构建适用于我的项目的 CNN。在我的项目中，我试图借助 FOMC 会议声明来预测 VIX 和标准普尔 500 指数。所以基本上我一方面使用文本数据，另一方面使用财务数据 (returns)。在预处理和应用 Googles Word2Vec pre-trained Word-Embeddings 之后，我构建了以下卷积网络：
def ConvNet(embeddings, max_sequence_length, num_words, embedding_dim, trainable=False, extra_conv=True,
lr=0.001, dropout=0.5):
embedding_layer = Embedding(num_words,
embedding_dim,
weights=[embeddings],
input_length=max_sequence_length,
trainable=trainable)
sequence_input = Input(shape=(max_sequence_length,), dtype='int32')
embedded_sequences = embedding_layer(sequence_input)
convs = []
filter_sizes = [3, 4, 5]
for filter_size in filter_sizes:
l_conv = Conv1D(filters=128, kernel_size=filter_size, activation='relu')(embedded_sequences)
l_pool = MaxPooling1D(pool_size=3)(l_conv)
convs.append(l_pool)
l_merge = concatenate([convs[0], convs[1], convs[2]], axis=1)
# add a 1D convnet with global maxpooling, instead of Yoon Kim model
conv = Conv1D(filters=128, kernel_size=3, activation='relu')(embedded_sequences)
pool = MaxPooling1D(pool_size=3)(conv)
if extra_conv == True:
x = Dropout(dropout)(l_merge)
else:
# Original Yoon Kim model
x = Dropout(dropout)(pool)
x = Flatten()(x)
x = Dense(128, activation='relu')(x)
preds = Dense(1, activation='linear')(x)
model = Model(sequence_input, preds)
sgd = SGD(learning_rate = lr, momentum= 0.8)
model.compile(loss='mean_squared_error',
optimizer= sgd,
metrics=['mean_squared_error'])
model.summary()
return model
我的模型架构是这样的：
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, 1086) 0
__________________________________________________________________________________________________
embedding_1 (Embedding) (None, 1086, 300) 532500 input_1[0][0]
__________________________________________________________________________________________________
conv1d_1 (Conv1D) (None, 1084, 128) 115328 embedding_1[0][0]
__________________________________________________________________________________________________
conv1d_2 (Conv1D) (None, 1083, 128) 153728 embedding_1[0][0]
__________________________________________________________________________________________________
conv1d_3 (Conv1D) (None, 1082, 128) 192128 embedding_1[0][0]
__________________________________________________________________________________________________
max_pooling1d_1 (MaxPooling1D) (None, 361, 128) 0 conv1d_1[0][0]
__________________________________________________________________________________________________
max_pooling1d_2 (MaxPooling1D) (None, 361, 128) 0 conv1d_2[0][0]
__________________________________________________________________________________________________
max_pooling1d_3 (MaxPooling1D) (None, 360, 128) 0 conv1d_3[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 1082, 128) 0 max_pooling1d_1[0][0]
max_pooling1d_2[0][0]
max_pooling1d_3[0][0]
__________________________________________________________________________________________________
dropout_2 (Dropout) (None, 1082, 128) 0 concatenate_1[0][0]
__________________________________________________________________________________________________
flatten_1 (Flatten) (None, 138496) 0 dropout_2[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 128) 17727616 flatten_1[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 1) 129 dense_3[0][0]
==================================================================================================
Total params: 18,721,429
Trainable params: 18,188,929
Non-trainable params: 532,500
所以，现在我面临下一个大问题，我真的运行不知道如何解决：超参数优化
我的问题是，到目前为止我发现的每个代码示例都应用于超参数的优化是架构：
model = Sequential()
embedding = model.add(layers.Embedding(MAX_VOCAB_SIZE, EMBEDDING_DIM, input_length=MAX_SEQUENCE_LENGTH))
model.add(layers.Conv1D(filters=128, kernel_size=5, activation='relu'))
model.add(layers.MaxPool1D(pool_size=3))
model.add(Dropout(0.25))
所以我的具体问题是，如何执行超参数优化，因为每当我更改我的 ConvNet 中的某些内容时，我都会收到错误，正如我所说，我能找到的所有教程都应用于 model = Sequential()。
新的错误信息是：
__________________________________________________________________________________________________
0%| | 0/100 [00:00<?, ?trial/s, best loss=?]
job exception: 'Model' object is not subscriptable
Traceback (most recent call last):
File "/Users/lukaskoston/Desktop/MasterarbeitFOMCAnalysis/07_Regression/CNN regression neu.py", line 262, in <module>
max_evals=100)
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 482, in fmin
show_progressbar=show_progressbar,
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/base.py", line 686, in fmin
show_progressbar=show_progressbar,
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 509, in fmin
rval.exhaust()
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 330, in exhaust
self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 286, in run
self.serial_evaluate()
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/fmin.py", line 165, in serial_evaluate
result = self.domain.evaluate(spec, ctrl)
File "/Users/lukaskoston/.local/lib/python3.7/site-packages/hyperopt/base.py", line 894, in evaluate
rval = self.fn(pyll_rval)
File "/Users/lukaskoston/Desktop/MasterarbeitFOMCAnalysis/07_Regression/CNN regression neu.py", line 248, in train_and_score
return hist['val_loss'][-1]
TypeError: 'Model' object is not subscriptable
提前致谢， 卢卡斯
您应该为您的方法设置超参数参数。
def ConvNet(embeddings, max_sequence_length, num_words, embedding_dim, trainable=False, extra_conv=True
lr=1.0, dropout=0.5):
# ...
然后您可以更新您的代码以使用这些值而不是固定值或 Keras 默认值。
if extra_conv == True:
x = Dropout(dropout)(l_merge)
else:
# Original Yoon Kim model
x = Dropout(dropout)(pool)
并且：
model.compile(loss='mean_squared_error',
optimizer=keras.optimizers.Adadelta(learning_rate=lr).
metrics=['mean_squared_error'])
我将从这两个开始，然后将批量大小和纪元计数留到以后。这些会对 运行 时间产生很大影响，这在超参数优化中很难解释。
然后您可以使用 hyperopt.
这样的库进行优化from hyperopt import fmin, hp, tpe, space_eval
def train_and_score(args):
# Train the model the fixed params plus the optimization args.
# Note that this method should return the final History object.
hist = ConvNet(embeddings, max_sequence_length, num_words,
embedding_dim, trainable=False, extra_conv=True,
lr=args['lr'], dropout=args['dropout'])
# Unpack and return the last validation loss from the history.
return hist['val_loss'][-1]
# Define the space to optimize over.
space = {
'lr': hp.loguniform('lr', np.log(0.1), np.log(10.0)),
'dropout': hp.uniform('dropout', 0, 1),
}
# Minimize the training score over the space.
trials = Trials()
best = fmin(train_and_score, space, trials=trials, algo=tpe.suggest, max_evals=100)
# Print details about the best results and hyperparameters.
print(best)
print(space_eval(space, best))
还有一些库可以帮助您直接将其与 Keras 集成。一个流行的选择是 hyperas。在那种情况下，您将修改您的函数以使用一些模板而不是参数，但它在其他方面非常相似。