TensorFlow 数据不适用于多输入 keras 模型

Question

我正在使用 TensorFlow Data 和 keras 构建管道来训练我的模型，直到这一刻一切都运行良好，使用单个输入图像。事实证明，我需要包含 2 个具有单独 CNN 架构的新图像，以便合并到单个神经网络中，总共 3 个输入。让我展示一下代码：

def read_tfrecord(example_proto):
    image_feature_description = {
    'label': tf.io.FixedLenFeature([], tf.int64),
    'full_image': tf.io.FixedLenFeature([], tf.string),
    'hand_0': tf.io.FixedLenFeature([], tf.string),
    'hand_1': tf.io.FixedLenFeature([], tf.string)
    }

    example = tf.io.parse_single_example(example_proto, image_feature_description)
    return read_full_image_and_label(example)

def load_dataset(tf_record_path):
    raw_dataset = tf.data.TFRecordDataset(tf_record_path, num_parallel_reads=AUTOTUNE)

    parsed_dataset = raw_dataset.map(read_tfrecord, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    return parsed_dataset 

def load_data_tfrecord(tfrecord_path, augmentation):
  dataset = load_dataset(tfrecord_path)

  dataset = prepare_for_training(dataset, augmentation)
  return dataset

data_generator['train'] = load_data_tfrecord(train_fns, True)

我按照这个顺序读取我的 TF 记录，直到此时代码与之前的代码几乎相同，添加了 'hand_0' 和 'hand_1'（我的新图像). 为了获取图像，我采用了这种方法：

def decode_and_resize_img(tf_btstring, img_width, img_height):
  image = tf.image.decode_jpeg(tf_btstring, channels=3)
  image = tf.image.resize(image, [img_width, img_height])

  return image 

def read_full_image_and_label(tf_bytestring):
    image = decode_and_resize_img(tf_bytestring['full_image'], 224, 224)
    hand_0 = decode_and_resize_img(tf_bytestring['hand_0'], 224, 224)
    hand_1 = decode_and_resize_img(tf_bytestring['hand_1'], 224, 224)  

    label = tf.cast(tf_bytestring['label'], tf.int32)

    return {'image_data': image, 'hnd_0': hand_0, 'hnd_1': hand_1}, label

我没有 return 只创建图像和标签，而是创建了一个对象来放置我所有的图像。在最后一步中，我准备了数据集来为模型提供数据：

def prepare_for_training(ds, augmentation=True, shuffle_buffer_size=1000):
    ds.cache()
    ds = ds.repeat()
    ds = ds.batch(batch_size)

    ds = ds.unbatch()
    ds = ds.shuffle(buffer_size=shuffle_buffer_size)
    ds = ds.batch(batch_size)
    ds = ds.prefetch(AUTOTUNE)
    return ds

在整个架构中，我只是通过一个对象和TFRecord描述改变了图像的return。我还更改了CNN/NN拓扑，如下：

    full_body_model = EfficientNetB4(input_shape=(img_width, img_height, 3), weights='imagenet', include_top=False)
    hand_1_model =  EfficientNetB0(input_shape=(224, 224, 3), weights='imagenet', include_top=False)
    hand_2_model =  EfficientNetB0(input_shape=(224, 224, 3), weights='imagenet', include_top=False)

    full_body_model.trainable = False
    hand_1_model.trainable = False
    hand_2_model.trainable = False

    for layer in hand_1_model.layers:
      layer._name = 'hand_1' + str(layer._name)

    for layer in hand_2_model.layers:
      layer._name = 'hand_2' + str(layer._name)

    fbm = full_body_model.output
    fbm = GlobalAveragePooling2D()(fbm)

    h1m = hand_1_model.output
    h1m = GlobalAveragePooling2D()(h1m)
    
    h2m = hand_2_model.output
    h2m = GlobalAveragePooling2D()(h2m)

    x = Concatenate()([fbm, h1m, h2m])
    x = Dense(512, activation='elu', name='fc1')(x)
    x = Dropout(0.4)(x)
    x = Dense(256, activation='elu', name='fc2')(x)
    x = Dropout(0.4)(x)
    x = Dense(128, activation='elu', name='fc3')(x)
    x = Dropout(0.4)(x)
    x = Dense(64, activation='elu', name='fc4')(x)
    x = Dropout(0.4)(x)

    predictions = Dense(nb_classes, activation='softmax', name='predictions')(x)

    model = Model([full_body_model.input, hand_1_model.input, hand_2_model.input], predictions)

现在，我遇到了这个错误：

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:811 train_function  *
        for _ in math_ops.range(self._steps_per_execution):
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:414 for_stmt
        symbol_names, opts)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:629 _tf_range_for_stmt
        opts)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:1059 _tf_while_stmt
        body, get_state, set_state, init_vars, nulls, symbol_names)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:1032 _try_handling_undefineds
        _verify_loop_init_vars(init_vars, symbol_names, first_iter_vars)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:193 _verify_loop_init_vars
        raise ValueError(error_msg)

    ValueError: 'outputs' must be defined before the loop.

我尝试过其他方法，例如此处：https://github.com/tensorflow/tensorflow/issues/20698，但我总是遇到同样的错误。我希望得到一些帮助来解决这个问题！

我使用的是 Google Colab 的当前版本，TF 2.4.0

Answer 1

为了解决这个问题，我改变了我的模型接收输入数据的方式。我正在使用自定义生成器将多输入图像堆叠在一起并将此列表传递给 fit 方法。这是我的代码：

    def train_gen():
      for (stacked, label) in train_ds:
        label = label 
        img1 = stacked[:,0,:,:]    
        img2 = stacked[:,1,:,:]
        img3 = stacked[:,2,:,:]
        img1, img2, img3 = transform_batch(img1, img2, img3) #data augmentation
        yield [img1, img2, img3], label

TensorFlow 数据不适用于多输入 keras 模型

TensorFlow Data not working with multiple input keras model

python

keras

tensorflow

tensorflow-datasets

tensorflow2.0