Tensorflow model.fit() 再现性

Question

import tensorflow as tf

RANDOM_SEED_CONSTANT = 42  # FOR_REPRODUCIBILITY
tf.random.set_seed(RANDOM_SEED_CONSTANT)

# Prevent NHWC errors https://www.nuomiphp.com/eplan/en/50125.html
from tensorflow.keras import backend as K
K.set_image_data_format("channels_last")

from tensorflow import keras
from tensorflow.keras import datasets, layers, models

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images, test_images = train_images / 255.0, test_images / 255.0 # Normalize pixel values to be between 0 and 1

# Create a simple CNN
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, 
                       activation='relu', 
                       kernel_initializer=tf.keras.initializers.HeNormal(seed=RANDOM_SEED_CONSTANT)))
model.add(layers.Dense(10, 
                       kernel_initializer=tf.keras.initializers.HeNormal(seed=RANDOM_SEED_CONSTANT)))

print(model.summary())

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.save_weights('myweights.h5')

# Run1
history = model.fit(train_images, train_labels, epochs=1, 
                    shuffle=False,
                    validation_data=(test_images, test_labels))

# Run2
model.load_weights('myweights.h5')
history = model.fit(train_images, train_labels, epochs=1, 
                    shuffle=False,
                    validation_data=(test_images, test_labels))

# Run3
model.load_weights('myweights.h5')
history = model.fit(train_images, train_labels, epochs=1, 
                    shuffle=False,
                    validation_data=(test_images, test_labels))

上面的 3 个 model.fit() 调用给我以下结果：

1563/1563 [==============================] - 7s 4ms/step - loss: 1.4939 - accuracy: 0.4543 - val_loss: 1.2516 - val_accuracy: 0.5567

1563/1563 [==============================] - 6s 4ms/step - loss: 1.6071 - accuracy: 0.4092 - val_loss: 1.3857 - val_accuracy: 0.4951

1563/1563 [==============================] - 7s 4ms/step - loss: 1.5538 - accuracy: 0.4325 - val_loss: 1.3187 - val_accuracy: 0.5294

造成这种差异的原因是什么？我试图了解可能阻碍从模型中再现结果的来源。除了随机种子，密集层初始化，我还缺少什么？

Answer 1

您测试再现性的方式不正确。你需要关闭程序重新运行看看结果是否一样。否则，运行 2 取决于运行 1 期间发生的事件，而运行 3 取决于运行 1 和 2 期间发生的事件。

原因是 Tensorflow 维护了一个用于随机生成的内部计数器，如 tf.random.set_seed 的文档中所述（重点是我的）：

print(tf.random.uniform([1], seed=1))  # generates 'A1'
print(tf.random.uniform([1], seed=1))  # generates 'A2'
The reason we get 'A2' instead 'A1' on the second call of tf.random.uniform above is because the same tf.random.uniform kernel (i.e. internal representation) is used by TensorFlow for all calls of it with the same arguments, and the kernel maintains an internal counter which is incremented every time it is executed, generating different results.

如果我运行只在程序的第一个运行两次，在每次运行之间关闭程序（在 IPython 中），我得到：

In [1]: run program.py
1563/1563 [==============================] - 13s 8ms/step - loss: 1.4997 - accuracy: 0.4540 - val_loss: 1.2528 - val_accuracy: 0.5494
{'loss': [1.4996991157531738], 'accuracy': [0.4540199935436249], 'val_loss': [1.2527965307235718], 'val_accuracy': [0.5493999719619751]}
In [2]: run program.py
1563/1563 [==============================] - 12s 8ms/step - loss: 1.4997 - accuracy: 0.4540 - val_loss: 1.2528 - val_accuracy: 0.5494
{'loss': [1.4996991157531738], 'accuracy': [0.4540199935436249], 'val_loss': [1.2527965307235718], 'val_accuracy': [0.5493999719619751]}

减去执行计算所花费的时间，这可能会因机器上的负载而有所不同，结果完全相同。

Tensorflow model.fit() 再现性

Tensorflow model.fit() reproducibility

deep-learning

keras

tensorflow

tensorflow2.x