Spark 在本地运行但不在 YARN 中运行

Question

我运行在本地模式就好了。当我在 YARN 模式下运行时，出现以下错误：

我收到这个错误：

 File "/hdfs15/yarn/nm/usercache/jvy234/filecache/11/spark-assembly-1.1.0-cdh5.2.1-hadoop2.5.0-cdh5.2.1.jar/pyspark/worker.py", line 79, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/hdfs15/yarn/nm/usercache/jvy234/filecache/11/spark-assembly-1.1.0-cdh5.2.1-hadoop2.5.0-cdh5.2.1.jar/pyspark/serializers.py", line 196, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/hdfs15/yarn/nm/usercache/jvy234/filecache/11/spark-assembly-1.1.0-cdh5.2.1-hadoop2.5.0-cdh5.2.1.jar/pyspark/serializers.py", line 127, in dump_stream
    for obj in iterator:
  File "/hdfs15/yarn/nm/usercache/jvy234/filecache/11/spark-assembly-1.1.0-cdh5.2.1-hadoop2.5.0-cdh5.2.1.jar/pyspark/serializers.py", line 185, in _batched
    for item in iterator:
  File "/home/jvy234/globalHawk.py", line 84, in <lambda>
TypeError: 'bool' object is not callable

        org.apache.spark.api.python.PythonRDD$$anon.read(PythonRDD.scala:124)
        org.apache.spark.api.python.PythonRDD$$anon.<init>(PythonRDD.scala:154)
        org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:87)
        org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:61)
        org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run.apply$mcV$sp(PythonRDD.scala:209)
        org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run.apply(PythonRDD.scala:184)
        org.apache.spark.api.python.PythonRDD$WriterThread$$anonfun$run.apply(PythonRDD.scala:184)
        org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1319)
        org.apache.spark.api.python.PythonRDD$WriterThread.run(PythonRDD.scala:183)

我脚本中的第 84 行是：

dataSplit = dataFile.map(lambda line: line.split(deli))

至运行本地：

spark-submit --master local globalHawk.py -i 20140817_011500_offer_init.dat -s kh_offers_schema4.txt4 -o txt.txt -d "|"

至运行 Yarn-Client：

spark-submit --master yarn-client globalHawk.py -i 20140817_011500_offer_init.dat -s kh_offers_schema4.txt4 -o txt.txt -d "|"

Answer 1

这个问题应该是由于驱动程序和 YARN worker 中 Python 的不同版本引起的，可以通过在 YARN 中的驱动程序和 worker 中使用相同版本的 Python 作为默认值来解决。

您还可以通过以下方式指定要在 YARN 中使用的 python 版本：

PYSPARK_PYTHON=python2.6 bin/spark-submit xxx

（没有可用的 YARN 集群，未测试）

Spark 在本地运行但不在 YARN 中运行

Spark Runs in Local but not in YARN

python

hadoop

apache-spark