Google Cloud Dataproc 删除 BigQuery table 不工作

Question

您好，我尝试使用 Dataproc 中的 java client library 从 BigQuery 中删除 table，如下所示启动了 spark-shell：

spark-shell --packages com.google.cloud:google-cloud-bigquery:1.59.0

并导入以下依赖项

import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.FieldValueList;
import com.google.cloud.bigquery.Job;
import com.google.cloud.bigquery.JobId;
import com.google.cloud.bigquery.JobInfo;
import com.google.cloud.bigquery.QueryJobConfiguration;
import com.google.cloud.bigquery.QueryResponse;
import com.google.cloud.bigquery.TableResult;
import java.util.UUID;

val bigquery = BigQueryOptions.getDefaultInstance().getService() 
bigquery.delete("test","temp")

这里的 test 和 temp 分别是我的数据集和 table 名称，但是在上面的运行语句之后它显示以下错误：

java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
  at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:82)
  at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:88)
  at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:74)
  at com.google.cloud.RetryHelper.run(RetryHelper.java:75)
  at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
  at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:386)
  at com.google.cloud.bigquery.BigQueryImpl.delete(BigQueryImpl.java:375)
  ... 48 elided

Answer 1

这是因为您在类路径上有一个没有 MoreExecutors.directExecutor 方法的旧 Guava library 版本（作为 Hadoop/Spark 依赖项引入）。

要解决此问题，您需要添加 shade/relocate (to avoid conflicts with other libraries on classpath) google-cloud-bigquery library and its dependencies (including Guava) into your application's UberJar.

Here is an example of how to do this using Maven Shade 插件。

Google Cloud Dataproc 删除 BigQuery table 不工作

Google Cloud Dataproc drop BigQuery table not working

google-bigquery

apache-spark

google-cloud-platform

google-cloud-dataproc