Spark结构化流kafka依赖无法解决

Spark structured streaming kafka dependency cannot be resolved

我试过了

./spark-2.3.1-bin-hadoop2.7/bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.1 test.py 

在我自己的电脑上,一切正常。但是在我在我学校的服务器上尝试之后,它有以下消息和错误。我在Google里找了半天也不知道。谁能帮帮我?

Ivy Default Cache set to: /home/zqwang/.ivy2/cache The jars for the packages stored in: /home/zqwang/.ivy2/jars :: loading settings :: url = jar:file:/data/opt/tmp/zqwang/spark-2.3.1-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.apache.spark#spark-sql-kafka-0-10_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent-26b526c6-0535-4007-8428-e38188af5709;1.0 confs: [default] :: resolution report :: resolve 966ms :: artifacts dl 0ms :: modules in use:


| | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded|


| default | 1 | 0 | 0 | 0 || 0 | 0 |


:: problems summary :: :::: WARNINGS module not found: org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1

==== local-m2-cache: tried

file:/home/zqwang/.m2/repository/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.pom

-- artifact

org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1!spark-sql-kafka-0-10_2.11.jar:

file:/home/zqwang/.m2/repository/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.jar

==== local-ivy-cache: tried

/home/zqwang/.ivy2/local/org.apache.spark/spark-sql-kafka-0-10_2.11/2.3.1/ivys/ivy.xml

-- artifact

org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1!spark-sql-kafka-0-10_2.11.jar:

/home/zqwang/.ivy2/local/org.apache.spark/spark-sql-kafka-0-10_2.11/2.3.1/jars/spark-sql-kafka-0-10_2.11.jar

==== central: tried

https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.pom

-- artifact

org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1!spark-sql-kafka-0-10_2.11.jar:

https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.jar

==== spark-packages: tried

http://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.pom

-- artifact

org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1!spark-sql-kafka-0-10_2.11.jar:

http://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.jar

  ::::::::::::::::::::::::::::::::::::::::::::::

  ::          UNRESOLVED DEPENDENCIES         ::

  ::::::::::::::::::::::::::::::::::::::::::::::

  :: org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1: not found

  ::::::::::::::::::::::::::::::::::::::::::::::

:::: ERRORS Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.pom (java.net.ConnectException: Connection refused)

Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.jar (java.net.ConnectException: Connection refused)

Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.pom (java.net.ConnectException: Connection refused)

Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.jar (java.net.ConnectException: Connection refused)

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: org.apache.spark#spark-sql-kafka-0-10_2.11;2.3.1: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1303) at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:53) at org.apache.spark.deploy.SparkSubmit$.doPrepareSubmitEnvironment(SparkSubmit.scala:364) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:250) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:171) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

But after i tried in my school's server, it has the following messages and errors

你的学校有防火墙阻止下载远程包。

这个 link 适合我,例如

Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.3.1/spark-sql-kafka-0-10_2.11-2.3.1.pom (java.net.ConnectException: Connection refused)

您需要在校外下载 Kafka jar,然后使用 --jars 标志与它们一起提交