Getting AttributeError: 'OneHotEncoder' object has no attribute '_jdf in pyspark'

Question

我正在尝试在 pyspark 的 kaggle 数据集上实现梯度提升算法以用于学习目的。我面临下面给出的错误

Traceback (most recent call last):
  File "C:/SparkCourse/Gradientboost.py", line 29, in <module>
    output=assembler.transform(data)
  File "C:\spark\python\lib\pyspark.zip\pyspark\ml\base.py", line 105, in transform
  File "C:\spark\python\lib\pyspark.zip\pyspark\ml\wrapper.py", line 281, in _transform
AttributeError: 'OneHotEncoder' object has no attribute '_jdf'

对应的代码是

from pyspark.sql import SparkSession
from pyspark.ml.feature import StringIndexer,VectorIndexer,OneHotEncoder,VectorAssembler





spark=SparkSession.builder.config("spark.sql.warehouse.dir", "file:///C:/temp").appName("Gradientboostapp").enableHiveSupport().getOrCreate()
data= spark.read.csv("C:/Users/codemen/Desktop/Timeseries Analytics/liver_patient.csv",header=True, inferSchema=True)
#data.show()
print(data.count())
#data.printSchema()
print("After deleting  null  values")

data=data.na.drop()
print(data.count())

data=StringIndexer(inputCol="Gender",outputCol="GenderIndex").fit(data)


#let onehot encode the data

data=OneHotEncoder(inputCol="GenderIndex",outputCol="gendervec")


usedfeature=["Age","gendervec","Total_Bilirubin","Direct_Bilirubin","Alkaline_Phosphotase","Alamine_Aminotransferase","Aspartate_Aminotransferase","Total_Protiens","Albumin","Albumin_and_Globulin_Ratio"]
#
assembler=VectorAssembler(inputCols=usedfeature,outputCol="features")
output=assembler.transform(data)
output.select("features","category").show()

我已经使用字符串索引器将性别类别转换为数字形式，然后我尝试对 Genderindex 值执行 OnehotEncoding。当我在代码中执行 VectorAssembler 时出现错误。我可能在这里错过了非常愚蠢的概念。请帮我弄清楚

Answer 1

这行代码不正确：data=OneHotEncoder(inputCol="GenderIndex",outputCol="gendervec")。您将 data 设置为等于 OneHotEncoder() 对象，而不是转换数据。您需要调用 transform 来对数据进行编码。它应该是这样的。

encoder=OneHotEncoder(inputCol="GenderIndex",outputCol="gendervec") data = encoder.transform(data)

Getting AttributeError: 'OneHotEncoder' object has no attribute '_jdf in pyspark'

Getting AttributeError: 'OneHotEncoder' object has no attribute '_jdf in pyspark'

pyspark

apache-spark-mllib