Tensorflow 不断打印与 FusedBatchNorm 相关的内容

Question

当我使用一个模型（pb 文件）进行推理时，tensorflow 会打印多行与 FusedBatchNorm 相关的代码，如下片段：

Optimizing fused batch norm node name: "FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2b_1x1/BatchNorm/FusedBatchNorm"
op: "FusedBatchNorm"
input: "FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2b_1x1/Conv2D"
input: "FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/gamma"
input: "FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/beta"
input: "FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/moving_mean"
input: "FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/moving_variance"
device: "/job:localhost/replica:0/task:0/device:GPU:0"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "data_format"
  value {
    s: "NHWC"
  }
}
attr {
  key: "epsilon"
  value {
    f: 0.001
  }
}
attr {
  key: "is_training"
  value {
    b: false
  }
}

推理结果还可以，就是有点慢。不知道FusedBatchNorm是什么东西，这家伙是不是拖慢了推理速度

我运行 Centos 7.2服务器上的模型，Nvidia Tesla P4(8vCPU, 32G内存, GPU内存7G)，一般一张图片需要12秒(我觉得真的很慢:-( ).

关于训练，我使用tensorflow对象检测和模型faster_rcnn_resnet101，这里是配置文件：

model {
  faster_rcnn {
    num_classes: 3
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension:  670
        max_dimension: 1013
      }
    }
    feature_extractor {
      type: "faster_rcnn_resnet101"
      first_stage_features_stride: 16
    }
    first_stage_anchor_generator {
      grid_anchor_generator {
        height_stride: 16
        width_stride: 16
        scales: 0.25
        scales: 0.5
        scales: 1.0
        scales: 2.0
        aspect_ratios: 0.5
        aspect_ratios: 1.0
        aspect_ratios: 2.0
      }
    }
    first_stage_box_predictor_conv_hyperparams {
      op: CONV
      regularizer {
        l2_regularizer {
          weight: 0.0
        }
      }
      initializer {
        truncated_normal_initializer {
          stddev: 0.01
        }
      }
    }
    first_stage_nms_score_threshold: 0.0
    first_stage_nms_iou_threshold: 0.7
    first_stage_max_proposals: 100
    first_stage_localization_loss_weight: 2.0
    first_stage_objectness_loss_weight: 1.0
    initial_crop_size: 14
    maxpool_kernel_size: 2
    maxpool_stride: 2
    second_stage_box_predictor {
      mask_rcnn_box_predictor {
        fc_hyperparams {
          op: FC
          regularizer {
            l2_regularizer {
              weight: 0.0
            }
          }
          initializer {
            variance_scaling_initializer {
              factor: 1.0
              uniform: true
              mode: FAN_AVG
            }
          }
        }
        use_dropout: false
        dropout_keep_probability: 1.0
      }
    }
    second_stage_post_processing {
      batch_non_max_suppression {
        score_threshold: 0.3
        iou_threshold: 0.6
        max_detections_per_class: 30
        max_total_detections: 30
      }
      score_converter: SOFTMAX
    }
    second_stage_localization_loss_weight: 2.0
    second_stage_classification_loss_weight: 1.0
  }
}
train_config {
  batch_size: 1
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  optimizer {
    momentum_optimizer {
      learning_rate {
        exponential_decay_learning_rate {
          initial_learning_rate: 0.0003
          decay_steps: 2000
          decay_factor: 0.95
        }
      }
      momentum_optimizer_value: 0.9
    }
    use_moving_average: false
  }
  gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "d:/od/tool/faster_rcnn3/model.ckpt"
  from_detection_checkpoint: true
}
train_input_reader {
  label_map_path: "d:/od/project/train_allinone/file/labelmap.pbtxt"
  tf_record_input_reader {
    input_path: "d:/od/project/train_allinone/file/tf.record"
  }
}

Answer 1

终于找到原因了：

我用于训练的桌面上的 Tensorflow 是 1.8，但在推理服务器上是 1.9。

抱歉打扰了。

Answer 2

关于详细的调试输出：看起来这是最近在主干中修复的错误：https://github.com/tensorflow/tensorflow/pull/19870

修复后，只有在 VLOG 级别设置为 2 或更高时才会打印输出。

Tensorflow 不断打印与 FusedBatchNorm 相关的内容

Tensorflow keep printing something related to FusedBatchNorm

python

object-detection

tensorflow