"ClassNotFoundException: sun.security.provider.Sun" 当 运行 Google Google App Engine 中的云数据流管道

Question

我们数据流管道中的一个 DoFn 包含一个类型，该类型具有指向 SecureRandom 实例的 Random 字段，并且当 运行 在使用 DataflowPipelineRunner 的数据流服务。 （下面的堆栈跟踪）

我们使用它的默认构造函数创建 SecureRandom，它恰好返回一个使用 sun.security.provider.Sun 作为它的 java.security.Provider 的实例（参见 SecureRandom#getProvider）。 SecureRandom 扩展了 Random，它是可序列化的。

Dataflow 服务在尝试反序列化此 class 时阻塞，因为它无法创建 sun.security.provider.Sun.

仔细查看堆栈跟踪，我发现反序列化是通过 com.google.apphosting.runtime.security.UserClassLoader 发生的，现在我的理论是这个 classloader 不允许加载 sun.* classes，或者至少这个特定的 sun.* class.

java.lang.IllegalArgumentException: unable to deserialize com.example.Example@13e88d
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:73)
    at com.google.cloud.dataflow.sdk.util.SerializableUtils.clone(SerializableUtils.java:88)
    at com.google.cloud.dataflow.sdk.transforms.ParDo$Bound.<init>(ParDo.java:683)
    [...]
    Caused by: java.lang.ClassNotFoundException: sun.security.provider.Sun
    at com.google.apphosting.runtime.security.UserClassLoader.loadClass(UserClassLoader.java:442)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:375)
    at java.lang.Class.forName0(Native Method)
    [...]

Answer 1

问题是 sun.security.provider.Sun 没有出现在 App Engine JRE 白名单中，因此 classloader 无法实例化它的实例：

https://cloud.google.com/appengine/docs/java/jrewhitelist

但幸运的是你仍然可以在相同的环境中说new SecureRandom()。

为了解决这个问题，我们添加了一个自定义 de/serialization 挂钩到 class 和 Random 字段。简化示例：

class Example implements Serializable {

  // See comments on {@link #writeObject} for why this is transient.
  // Should be treated as final, but can't be declared as such.
  private transient Random random;

  //
  // [Guts of the class go here...]
  //

  /**
   * Serialization hook to handle the transient Random field.
   */
  private void writeObject(ObjectOutputStream out) throws IOException {
    out.defaultWriteObject();
    if (random instanceof SecureRandom) {
      // Write a null to tell readObject() to create a new
      // SecureRandom during deserialization; null is safe to use
      // as a placeholder because the constructor disallows null
      // Randoms.
      //
      // The dataflow cloud environment won't deserialize
      // SecureRandom instances that use sun.security.provider.Sun
      // as their Provider, because it's a system
      // class that's not on the App Engine whitelist:
      // https://cloud.google.com/appengine/docs/java/jrewhitelist
      out.writeObject(null);
    } else {
      out.writeObject(random);
    }
  }

  /**
   * Deserialization hook to initialize the transient Random field.
   */
  private void readObject(ObjectInputStream in)
      throws IOException, ClassNotFoundException {
    in.defaultReadObject();
    Object newRandom = in.readObject();
    if (newRandom == null) {
      // writeObject() will write a null if the original field was
      // SecureRandom; create a new instance to replace it. See
      // comments in writeObject() for background.
      random = new SecureRandom();
      random.nextDouble(); // force seeding
    } else {
      random = (Random) newRandom;
    }
  }
}