使用选项启动 pyspark 时出错（没有 Spack 包）

Question

谁能告诉我为什么会出现以下错误？根据 pyspark-cassandra 连接器的自述文件，我在下面尝试的应该可以工作（没有 Spark 包）：https://github.com/TargetHolding/pyspark-cassandra

$ pyspark_jar="$HOME/devel/sandbox/Learning/Spark/pyspark-cassandra/target/scala-2.10/pyspark-cassandra-assembly-0.2.2.jar"

$ pyspark_egg="$HOME/devel/sandbox/Learning/Spark/pyspark-cassandra/target/pyspark_cassandra-0.2.2-py2.7.egg"
$ pyspark --jars $pyspark_jar --py_files $pyspark_egg --conf spark.cassandra.connection.host=localhost

这导致：

Exception in thread "main" java.lang.IllegalArgumentException: pyspark does not support any application options.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:222)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:239)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:113)
at org.apache.spark.launcher.Main.main(Main.java:74)

Answer 1

解决了问题。我需要使用

--py-files

而不是

--py_files

使用选项启动 pyspark 时出错（没有 Spack 包）

Error starting pyspark with options (Without Spack packages)

apache-spark

pyspark