使用具有 Keras/Tensorflow 的稀疏数据生成器

Using sparse data generator with Keras/Tensorflow

我已经使用 CPU 在 C++ 中实现了一个网络,我正在尝试使用 GPU 和 python 来训练它。我面临的问题是输入非常大(而且稀疏),大约有 50000 个输入神经元,通常只有 30 个被激活。

我的模型是这样的:

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            (None, 24576)        0                                            
__________________________________________________________________________________________________
input_2 (InputLayer)            (None, 24576)        0                                            
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 256)          6291712     input_1[0][0]                    
__________________________________________________________________________________________________
dense_2 (Dense)                 (None, 256)          6291712     input_2[0][0]                    
__________________________________________________________________________________________________
leaky_re_lu_1 (LeakyReLU)       (None, 256)          0           dense_1[0][0]                    
__________________________________________________________________________________________________
leaky_re_lu_2 (LeakyReLU)       (None, 256)          0           dense_2[0][0]                    
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 512)          0           leaky_re_lu_1[0][0]              
                                                                 leaky_re_lu_2[0][0]              
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 32)           16416       concatenate_1[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_3 (LeakyReLU)       (None, 32)           0           dense_3[0][0]                    
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 32)           1056        leaky_re_lu_3[0][0]              
__________________________________________________________________________________________________
leaky_re_lu_4 (LeakyReLU)       (None, 32)           0           dense_4[0][0]                    
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 1)            33          leaky_re_lu_4[0][0]              
==================================================================================================
Total params: 12,600,929
Trainable params: 12,600,929
Non-trainable params: 0

我还得到了大约 3 亿 input/output 个值,我正试图将其输入我的网络。 不用说,数据太多了,我的 GPU 一下子装不下。

出于速度目的,我生成了稀疏矩阵,每个矩阵代表大约 100000 个输入并将它们保存在内存中(大约 50Gb)。我可以像这样轻松地加载它们而不会损失太多速度:

# loads both the inputs and the output for the given chunk (100000 inputs/outputs) from the memory
trainX1,trainX2,trainY = readNumpyChunkAndCreateInput(chunk)

我用它来训练我的网络:

for chunk in chunks:
        trainX1,trainX2,trainY = readNumpyChunkAndCreateInput(chunk)

        _res = model.fit([trainX1,trainX2], trainY, epochs=1,steps_per_epoch=1,verbose=0)
        loss = list(_res.history.values())[0]
        totalLoss += loss[0]

显然这无论如何都不是最优的。我知道 Keras/TensorFlow 中有一个叫做 data generators 的东西,但遗憾的是我不知道如何在我的特定情况下使用它们,因为所有教程都处理密集输入。 如果有人能帮助我,我将非常高兴!

您好, 芬恩

编辑 1

我加载数据的方式:

filePath = os.path.abspath(os.path.dirname(sys.argv[0]))
    path = filePath + "\data\" + name + "\"

    indices1 = np.load(path + 'indices1.npy')
    indices2 = np.load(path + 'indices2.npy')
    outputs = np.load(path + 'outputs.npy')

    meta = open(path + 'meta.txt', "r")
    metaInf = meta.readlines()[0].split(" ")
    meta.close()

    entry1Count = int(metaInf[0])
    entry2Count = int(metaInf[1])
    lineCount = int(metaInf[2])

    values1 = tf.ones(entry1Count)
    values2 = tf.ones(entry2Count)

    shape = (lineCount, 6 * 64 * 64)

    trainX1 = tf.SparseTensor(
        indices=indices1,
        values=values1,
        dense_shape=shape
    )

    trainX2 = tf.SparseTensor(
        indices=indices2,
        values=values2,
        dense_shape=shape
    )

    return trainX1, trainX2, outputs

我已经编写了一个小型生成器函数,您可以根据自己的用例进行调整。

import os
def gen():
    paths = os.listdir('temp_data') # path of the directory
    for path in paths:
        file_path = os.path.join('temp_data',path)
        x = np.load(file_path)
        y = np.load(file_path),
        z = np.load(file_path)
        # Your logic
        #
        #
        #
        
        yield (x,y,z) # Three tensors/numpy arrays. In your case trainx1, trainx2, outputs.

在tf.data.Dataset中使用生成器的代码:

dataset = tf.data.Dataset.from_generator(gen, (tf.float32, tf.float32,tf.float32))
dataset = dataset.prefetch(2)

预取允许提前存储下一批,以消除任何延迟。 您可以使用此数据集传递给您的拟合命令或像这样使用自定义训练循环。

epochs = 100
for epoch in range(epochs):
    print("\nStart of epoch %d" % (epoch,))

    # Iterate over the batches of the dataset.
    for step, (x1_batch_train, x2_batch_train, y_batch_train) in enumerate(train_dataset):

        # Open a GradientTape to record the operations run
        # during the forward pass, which enables auto-differentiation.
        with tf.GradientTape() as tape:

            # Run the forward pass of the layer.
            # The operations that the layer applies
            # to its inputs are going to be recorded
            # on the GradientTape.
            logits = model([x1_batch_train,x2_batch_train], training=True)  # Logits for this minibatch

            # Compute the loss value for this minibatch.
            loss_value = loss_fn(y_batch_train, logits)

        # Use the gradient tape to automatically retrieve
        # the gradients of the trainable variables with respect to the loss.
        grads = tape.gradient(loss_value, model.trainable_weights)

        # Run one step of gradient descent by updating
        # the value of the variables to minimize the loss.
        optimizer.apply_gradients(zip(grads, model.trainable_weights))

        # Log every 200 batches.
        if step % 200 == 0:
            print(
                "Training loss (for one batch) at step %d: %.4f"
                % (step, float(loss_value))
            )
            print("Seen so far: %s samples" % ((step + 1) * 64))