Tensorflow.js 裁剪图像 return 零张量

Question

我有一个自定义模型，它接受来自 BlazeFace 模型的裁剪面孔，然后输出 3 类的预测。

在将它们发送到我的自定义模型之前，我将裁剪后的脸调整为 [1,224,224,3]

的形状

每次预测的输出：

Float32Array [
  6.522771905936864e-11,
  3.698188456857654e-12,
  1,
]

调整裁剪后的人脸大小并进行预测的代码：

 const getPrediction = async tensor => {
    if (!tensor) {
      console.log("Tensor not found!");
    }
    // Load both models 
    const bfModel = await blazeFaceModel;
    const returnTensors = true;
    const faces = await bfModel
      .estimateFaces(tensor, returnTensors)
      .catch(e => console.log(e));
    // Reshape tensor from rank 3 to rank 4
    const tensorReshaped = tensor.reshape([1, 224, 224, 3]);
    const scale = {
      height: styles.camera.height / tensorDims.height,
      width: styles.camera.width / tensorDims.width
    };

    // Faces is an array of objects
    if (!isEmpty(faces)) {
      // Setting faces in state
      setModelFaces({ faces });
    } 
    //Looping over array of objects in faces
    faces.map((face, i) => {
      const { topLeft, bottomRight } = face;
      const width = Math.floor(
        bottomRight.dataSync()[0] - topLeft.dataSync()[0] * scale.width
      );
      const height = Math.floor(
        bottomRight.dataSync()[1] - topLeft.dataSync()[1] * scale.height
      );
      const boxes = tf
        .concat([topLeft.dataSync(), bottomRight.dataSync()])
        .reshape([-1, 4]); 
      // Cropping out faces from original tensor 
      const crop = tf.image.cropAndResize(
        tensorReshaped,
        boxes,
        [0],
        [height, width]
      );
      // Resize cropped faces to [1,224,224,3]
      const alignCorners = true;
      const imageResize = tf.image.resizeBilinear(
        crop,
        [224, 224],
        alignCorners
      );
      makePrediction(imageResize);
    });
  };
// Make predictions on the tensors
  const makePrediction = async image => {
    if (!image) {
      console.log("No input!");
    }
    const model = await loadedModel;
    const prediction = await model.predict(image, { batchSize: 1 });
    if (!prediction || isEmpty(prediction)) {
      console.log("Prediction not available");
    }
    console.log(prediction);
    console.log(prediction.dataSync());
  };

编辑

我尝试将预测的批量大小更改为 1，但问题仍然存在

我尝试将 keras 模型重新转换为 tfjs 格式，但仍然是同样的问题

我在预测后尝试处理张量，但仍然存在错误

所以我打印出调整大小后的面的张量及其很多 0

Tensor before prediction
Tensor
    [[[[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      ...
      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]]]]
undefined
Tensor during prediction
Tensor
    [[[[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      ...
      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]],

      [[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0],
       ...,
       [0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]]]]
undefined

Answer 1

boxes of tf.image.cropAndResize 是0到1之间的归一化坐标。因此topLeft和bottomRight应该使用[imageWidth, imageHeight]

归一化

normalizedTopLeft = topLeft.div(tensor.shape.slice(-3, -2)) 
// slice will get [h, w] of a tensor of shape [b, h, w, ch] or [h, w, ch]

// do likewise for bottomRight
// use normalizedTopLeft instead of topLeft for cropping

Tensorflow.js 裁剪图像 return 零张量

Tensorflow.js cropping image return zeros tensor

reactjs

deep-learning

keras

tensorflow

tensorflow.js