Apache Beam 运行 与 Flink 抛出 NoSuchMethodError
Apache Beam run with Flink throws NoSuchMethodError
我通过 maven 得到了一个 beam 程序 jar，我想 运行 它与 flink 本地。
当我 运行 这样的时候，没关系:
mvn exec:java -Dexec.mainClass=GroupbyTest -Dexec.args="--runner=FlinkRunner
--flinkMaster=localhost:6123
--filesToStage=target/beamTest-1.0-SNAPSHOT.jar"
但是当我使用 flink 运行 时，protobuf 出现了问题：
./bin/flink run /home/maqy/Documents/beam_samples/beamTest/target/beamTest-1.0-SNAPSHOT.jar --runner=FlinkRunner
并且有日志：
Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop-2.7.5/etc/hadoop:/usr/local/hadoop-2.7.5/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/common/*:/usr/local/hadoop-2.7.5/share/hadoop/hdfs:/usr/local/hadoop-2.7.5/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.5/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/yarn/*:/usr/local/hadoop-2.7.5/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/mapreduce/*:/usr/local/hadoop-2.7.5/contrib/capacity-scheduler/*.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/maqy/%e4%b8%8b%e8%bd%bd/flink-1.4.0/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8081
Starting execution of program
------------------------------------------------------------
The program finished with the following exception:
java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.<init>(GeneratedMessageV3.java:1707)
at com.google.protobuf.AnyProto.<clinit>(AnyProto.java:52)
at org.apache.beam.model.pipeline.v1.RunnerApi.<clinit>(RunnerApi.java:53271)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$TransformsDefaultEntryHolder.<clinit>(RunnerApi.java:448)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.internalGetTransforms(RunnerApi.java:1339)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.getTransformsOrDefault(RunnerApi.java:1404)
at org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:81)
at org.apache.beam.runners.core.construction.PipelineTranslation.visitPrimitiveTransform(PipelineTranslation.java:87)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:670)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access0(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:53)
at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:91)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:110)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at GroupbyTest.main(GroupbyTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:417)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:396)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:802)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:282)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1054)
at org.apache.flink.client.CliFrontend.call(CliFrontend.java:1101)
at org.apache.flink.client.CliFrontend.call(CliFrontend.java:1098)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1098)
我想知道如何修复它，谢谢。
A NoSuchMethodError 通常表示版本问题。在 class 路径中找到了具有正确包和名称的 class（否则您会看到 ClassNotFoundException），但它缺少某种方法。
如果 class 路径包含错误的依赖项版本，通常会发生这种情况，在您的情况下可能是 Google ProtoBuf。
我已经使用 Beam 2.6 解决了这个问题。
我通过 maven 得到了一个 beam 程序 jar，我想 运行 它与 flink 本地。 当我 运行 这样的时候，没关系:
mvn exec:java -Dexec.mainClass=GroupbyTest -Dexec.args="--runner=FlinkRunner
--flinkMaster=localhost:6123
--filesToStage=target/beamTest-1.0-SNAPSHOT.jar"
但是当我使用 flink 运行 时，protobuf 出现了问题：
./bin/flink run /home/maqy/Documents/beam_samples/beamTest/target/beamTest-1.0-SNAPSHOT.jar --runner=FlinkRunner
并且有日志：
Using the result of 'hadoop classpath' to augment the Hadoop classpath: /usr/local/hadoop-2.7.5/etc/hadoop:/usr/local/hadoop-2.7.5/share/hadoop/common/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/common/*:/usr/local/hadoop-2.7.5/share/hadoop/hdfs:/usr/local/hadoop-2.7.5/share/hadoop/hdfs/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/hdfs/*:/usr/local/hadoop-2.7.5/share/hadoop/yarn/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/yarn/*:/usr/local/hadoop-2.7.5/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-2.7.5/share/hadoop/mapreduce/*:/usr/local/hadoop-2.7.5/contrib/capacity-scheduler/*.jar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/maqy/%e4%b8%8b%e8%bd%bd/flink-1.4.0/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.7.5/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Cluster configuration: Standalone cluster with JobManager at localhost/127.0.0.1:6123
Using address localhost:6123 to connect to JobManager.
JobManager web interface address http://localhost:8081
Starting execution of program
------------------------------------------------------------
The program finished with the following exception:
java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.<init>(GeneratedMessageV3.java:1707)
at com.google.protobuf.AnyProto.<clinit>(AnyProto.java:52)
at org.apache.beam.model.pipeline.v1.RunnerApi.<clinit>(RunnerApi.java:53271)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$TransformsDefaultEntryHolder.<clinit>(RunnerApi.java:448)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.internalGetTransforms(RunnerApi.java:1339)
at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.getTransformsOrDefault(RunnerApi.java:1404)
at org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:81)
at org.apache.beam.runners.core.construction.PipelineTranslation.visitPrimitiveTransform(PipelineTranslation.java:87)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:670)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access0(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59)
at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:53)
at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:91)
at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:110)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
at GroupbyTest.main(GroupbyTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:417)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:396)
at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:802)
at org.apache.flink.client.CliFrontend.run(CliFrontend.java:282)
at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1054)
at org.apache.flink.client.CliFrontend.call(CliFrontend.java:1101)
at org.apache.flink.client.CliFrontend.call(CliFrontend.java:1098)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1098)
我想知道如何修复它，谢谢。
A NoSuchMethodError 通常表示版本问题。在 class 路径中找到了具有正确包和名称的 class（否则您会看到 ClassNotFoundException），但它缺少某种方法。
如果 class 路径包含错误的依赖项版本，通常会发生这种情况，在您的情况下可能是 Google ProtoBuf。
我已经使用 Beam 2.6 解决了这个问题。