如何在 pyspark 管道阶段处理 stringindexer 和 onehotencoder

Question

遇到此代码的错误：

stage_string = [StringIndexer(inputCol=c, outputCol=c + "_string_encoded")  for c in categorical_columns]
stage_one_hot = [OneHotEncoder(inputCol=c + "_string_encoded", outputCol=c + "_one_hot") for c in categorical_columns]
assembler = VectorAssembler(inputCols=feature_list, outputCol="features")
rf = RandomForestClassifier(labelCol="output", featuresCol="features")
pipeline = Pipeline(stages=[stage_string,stage_one_hot,assembler, rf]) 
pipeline.fit(df)

Cannot recognize a pipeline stage of type <class 'list'>.
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/base.py", line 132, in fit
    return self._fit(dataset)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/ml/pipeline.py", line 97, in _fit
    "Cannot recognize a pipeline stage of type %s." % type(stage))
TypeError: Cannot recognize a pipeline stage of type <class 'list'>.

Answer 1

这个 pipeline = Pipeline(stages=[stage_string,stage_one_hot,assembler, rf]) 语句有问题 stage_string 和 stage_one_hot 是 PipelineStage 和 [=15= 的列表] 和 rf 是单独的流水线阶段。

修改你的声明如下-

stages = stage_string + stage_one_hot + [assembler, rf]
pipeline = Pipeline(stages=stages)

如何在 pyspark 管道阶段处理 stringindexer 和 onehotencoder

how to handle string indexer and onehot encoder in pyspark pipeline stages

python

pipeline

machine-learning

pyspark