无法提交 spark python 脚本

Question

我正在使用以下脚本提交 python 脚本

#!/usr/bin/python

from pyspark.mllib.classification import LogisticRegressionWithSGD
from pyspark.mllib.regression import LabeledPoint
from numpy import array
from pyspark import SparkContext as sc, SparkConf

data = sc.textFile("hdfs:/dataset/parkinsons.data")

收到此错误：

data = sc.textFile("hdfs:/dataset/parkinsons.data")
TypeError: unbound method textFile() must be called with SparkContext instance as first argument (got str instance instead)

Answer 1

您必须先创建一个SparkContext，例如：

from pyspark import SparkContext

sc = SparkContext(appName="TestApp")
data = sc.textFile("hdfs:/dataset/parkinsons.data")