自定义 Sklearn Transformer 单独工作，在管道中使用时抛出错误
Custom Sklearn Transformer works alone, Throws Error When Used in Pipeline
我有一个简单的 sklearn class 我想用作 sklearn 管道的一部分。这个 class 只需要一个 pandas 数据帧 X_DF 和一个分类列名，并调用 pd.get_dummies 到 return 数据帧，该列变成一个虚拟矩阵变量...
import pandas as pd
from sklearn.base import TransformerMixin, BaseEstimator
class dummy_var_encoder(TransformerMixin, BaseEstimator):
'''Convert selected categorical column to (set of) dummy variables
'''
def __init__(self, column_to_dummy='default_col_name'):
self.column = column_to_dummy
print self.column
def fit(self, X_DF, y=None):
return self
def transform(self, X_DF):
''' Update X_DF to have set of dummy-variables instead of orig column'''
# convert self-attribute to local var for ease of stepping through function
column = self.column
# add columns for new dummy vars, and drop original categorical column
dummy_matrix = pd.get_dummies(X_DF[column], prefix=column)
new_DF = pd.concat([X_DF[column], dummy_matrix], axis=1)
return new_DF
现在单独使用这个转换器 fit/transform，我得到了预期的输出。部分玩具数据如下：
from sklearn import datasets
# Load toy data
iris = datasets.load_iris()
X = pd.DataFrame(iris.data, columns = iris.feature_names)
y = pd.Series(iris.target, name='y')
# Create Arbitrary categorical features
X['category_1'] = pd.cut(X['sepal length (cm)'],
bins=3,
labels=['small', 'medium', 'large'])
X['category_2'] = pd.cut(X['sepal width (cm)'],
bins=3,
labels=['small', 'medium', 'large'])
...我的虚拟编码器产生正确的输出：
encoder = dummy_var_encoder(column_to_dummy = 'category_1')
encoder.fit(X)
encoder.transform(X).iloc[15:21,:]
category_1
category_1 category_1_small category_1_medium category_1_large
15 medium 0 1 0
16 small 1 0 0
17 small 1 0 0
18 medium 0 1 0
19 small 1 0 0
20 small 1 0 0
但是，当我从如下定义的 sklearn 管道调用同一个转换器时：
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import KFold, GridSearchCV
# Define Pipeline
clf = LogisticRegression(penalty='l1')
pipeline_steps = [('dummy_vars', dummy_var_encoder()),
('clf', clf)
]
pipeline = Pipeline(pipeline_steps)
# Define hyperparams try for dummy-encoder and classifier
# Fit 4 models - try dummying category_1 vs category_2, and using l1 vs l2 penalty in log-reg
param_grid = {'dummy_vars__column_to_dummy': ['category_1', 'category_2'],
'clf__penalty': ['l1', 'l2']
}
# Define full model search process
cv_model_search = GridSearchCV(pipeline,
param_grid,
scoring='accuracy',
cv = KFold(),
refit=True,
verbose = 3)
在我安装管道之前一切正常，此时我从虚拟编码器收到错误：
cv_model_search.fit(X,y=y)
In [101]: cv_model_search.fit(X,y=y) Fitting 3 folds for each of 4
candidates, totalling 12 fits
None None None None
[CV] dummy_vars__column_to_dummy=category_1, clf__penalty=l1 .........
Traceback (most recent call last):
File "", line 1, in
cv_model_search.fit(X,y=y)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/model_selection/_search.py",
line 638, in fit
cv.split(X, y, groups)))
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py",
line 779, in call
while self.dispatch_one_batch(iterator):
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py",
line 625, in dispatch_one_batch
self._dispatch(tasks)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py",
line 588, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.py",
line 111, in apply_async
result = ImmediateResult(func)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.py",
line 332, in init
self.results = batch()
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py",
line 131, in call
return [func(*args, **kwargs) for func, args, kwargs in self.items]
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/model_selection/_validation.py",
line 437, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/pipeline.py",
line 257, in fit
Xt, fit_params = self._fit(X, y, **fit_params)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/pipeline.py",
line 222, in _fit
**fit_params_steps[name])
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/memory.py",
line 362, in call
return self.func(*args, **kwargs)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/pipeline.py",
line 589, in _fit_transform_one
res = transformer.fit_transform(X, y, **fit_params)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/base.py",
line 521, in fit_transform
return self.fit(X, y, **fit_params).transform(X)
File "", line 21, in transform
dummy_matrix = pd.get_dummies(X_DF[column], prefix=column)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/frame.py",
line 1964, in getitem
return self._getitem_column(key)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/frame.py",
line 1971, in _getitem_column
return self._get_item_cache(key)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/generic.py",
line 1645, in _get_item_cache
values = self._data.get(item)
File
"/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/internals.py",
line 3599, in get
raise ValueError("cannot label index with a null key")
ValueError: cannot label index with a null key
跟踪告诉您到底出了什么问题。学习诊断跟踪确实非常宝贵，尤其是当您从您可能不完全了解的库继承时。
现在，我自己在 sklearn 中做了一些继承工作，我可以毫无疑问地告诉你 GridSearchCV 如果输入 fit 的数据类型会给你带来一些麻烦] 或 fit_transform 方法不是 NumPy 数组。正如 Vivek 在他的评论中提到的，传递给 fit 方法的 X 不再是 DataFrame。不过我们还是先看看trace吧
ValueError: cannot label index with a null key
虽然 Vivek 对 NumPy 数组的看法是正确的，但您这里还有另一个问题。您得到的实际错误是 fit 方法中 column 的值是 None。如果您查看上面的 encoder 对象，您会看到 __repr__ 方法输出以下内容：
dummy_var_encoder(column_to_dummy=None)
使用 Pipeline 时，此参数会被初始化并传递给 GridSearchCV。这种行为也可以在交叉验证和搜索方法中看到，并且具有与输入参数不同名称的属性会导致此类问题。解决此问题将使您走上正确的道路。
这样修改 __init__ 方法将解决这个特定问题：
def __init__(self, column='default_col_name'):
self.column = column
print(self.column)
但是，一旦完成此操作，Vivek 提到的问题就会浮出水面，您将不得不处理它。这是我之前 运行 了解的内容，尽管不是专门针对 DataFrames。我在 Use sklearn GridSearchCV on custom class whose fit method takes 3 arguments 中想出了一个解决方案。基本上，我创建了一个实现 __getitem__ 方法的包装器，使数据的外观和行为方式能够通过 GridSearchCV、Pipeline 和其他方法中使用的验证方法交叉验证方法。
编辑
我进行了这些更改，看来您的问题来自验证方法 check_array。虽然使用 dtype=pd.DataFrame 调用此方法会起作用，但线性模型使用 dtype=np.float64 调用此方法会抛出错误。要解决这个问题，而不是将原始数据与你的虚拟数据连接起来，你可以 return 你的虚拟列并使用它们进行拟合。这是无论如何都应该做的事情，因为您不想在您尝试拟合的模型中同时包含虚拟列和原始数据。您也可以考虑 drop_first 选项，但我要跑题了。因此，像这样更改您的 fit 方法可以让整个过程按预期工作。
def transform(self, X_DF):
''' Update X_DF to have set of dummy-variables instead of orig column'''
# convert self-attribute to local var for ease of stepping through function
column = self.column
# add columns for new dummy vars, and drop original categorical column
dummy_matrix = pd.get_dummies(X_DF[column], prefix=column)
return dummy_matrix
我有一个简单的 sklearn class 我想用作 sklearn 管道的一部分。这个 class 只需要一个 pandas 数据帧 X_DF 和一个分类列名，并调用 pd.get_dummies 到 return 数据帧，该列变成一个虚拟矩阵变量...
import pandas as pd
from sklearn.base import TransformerMixin, BaseEstimator
class dummy_var_encoder(TransformerMixin, BaseEstimator):
'''Convert selected categorical column to (set of) dummy variables
'''
def __init__(self, column_to_dummy='default_col_name'):
self.column = column_to_dummy
print self.column
def fit(self, X_DF, y=None):
return self
def transform(self, X_DF):
''' Update X_DF to have set of dummy-variables instead of orig column'''
# convert self-attribute to local var for ease of stepping through function
column = self.column
# add columns for new dummy vars, and drop original categorical column
dummy_matrix = pd.get_dummies(X_DF[column], prefix=column)
new_DF = pd.concat([X_DF[column], dummy_matrix], axis=1)
return new_DF
现在单独使用这个转换器 fit/transform，我得到了预期的输出。部分玩具数据如下：
from sklearn import datasets
# Load toy data
iris = datasets.load_iris()
X = pd.DataFrame(iris.data, columns = iris.feature_names)
y = pd.Series(iris.target, name='y')
# Create Arbitrary categorical features
X['category_1'] = pd.cut(X['sepal length (cm)'],
bins=3,
labels=['small', 'medium', 'large'])
X['category_2'] = pd.cut(X['sepal width (cm)'],
bins=3,
labels=['small', 'medium', 'large'])
...我的虚拟编码器产生正确的输出：
encoder = dummy_var_encoder(column_to_dummy = 'category_1')
encoder.fit(X)
encoder.transform(X).iloc[15:21,:]
category_1
category_1 category_1_small category_1_medium category_1_large
15 medium 0 1 0
16 small 1 0 0
17 small 1 0 0
18 medium 0 1 0
19 small 1 0 0
20 small 1 0 0
但是，当我从如下定义的 sklearn 管道调用同一个转换器时：
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import KFold, GridSearchCV
# Define Pipeline
clf = LogisticRegression(penalty='l1')
pipeline_steps = [('dummy_vars', dummy_var_encoder()),
('clf', clf)
]
pipeline = Pipeline(pipeline_steps)
# Define hyperparams try for dummy-encoder and classifier
# Fit 4 models - try dummying category_1 vs category_2, and using l1 vs l2 penalty in log-reg
param_grid = {'dummy_vars__column_to_dummy': ['category_1', 'category_2'],
'clf__penalty': ['l1', 'l2']
}
# Define full model search process
cv_model_search = GridSearchCV(pipeline,
param_grid,
scoring='accuracy',
cv = KFold(),
refit=True,
verbose = 3)
在我安装管道之前一切正常，此时我从虚拟编码器收到错误：
cv_model_search.fit(X,y=y)
In [101]: cv_model_search.fit(X,y=y) Fitting 3 folds for each of 4 candidates, totalling 12 fits
None None None None [CV] dummy_vars__column_to_dummy=category_1, clf__penalty=l1 .........
Traceback (most recent call last):
File "", line 1, in cv_model_search.fit(X,y=y)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/model_selection/_search.py", line 638, in fit cv.split(X, y, groups)))
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 779, in call while self.dispatch_one_batch(iterator):
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 625, in dispatch_one_batch self._dispatch(tasks)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 588, in _dispatch job = self._backend.apply_async(batch, callback=cb)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 111, in apply_async result = ImmediateResult(func)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 332, in init self.results = batch()
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 131, in call return [func(*args, **kwargs) for func, args, kwargs in self.items]
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/model_selection/_validation.py", line 437, in _fit_and_score estimator.fit(X_train, y_train, **fit_params)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/pipeline.py", line 257, in fit Xt, fit_params = self._fit(X, y, **fit_params)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/pipeline.py", line 222, in _fit **fit_params_steps[name])
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/externals/joblib/memory.py", line 362, in call return self.func(*args, **kwargs)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/pipeline.py", line 589, in _fit_transform_one res = transformer.fit_transform(X, y, **fit_params)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/sklearn/base.py", line 521, in fit_transform return self.fit(X, y, **fit_params).transform(X)
File "", line 21, in transform dummy_matrix = pd.get_dummies(X_DF[column], prefix=column)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/frame.py", line 1964, in getitem return self._getitem_column(key)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/frame.py", line 1971, in _getitem_column return self._get_item_cache(key)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/generic.py", line 1645, in _get_item_cache values = self._data.get(item)
File "/home/max/anaconda3/envs/remine/lib/python2.7/site-packages/pandas/core/internals.py", line 3599, in get raise ValueError("cannot label index with a null key")
ValueError: cannot label index with a null key
跟踪告诉您到底出了什么问题。学习诊断跟踪确实非常宝贵，尤其是当您从您可能不完全了解的库继承时。
现在，我自己在 sklearn 中做了一些继承工作，我可以毫无疑问地告诉你 GridSearchCV 如果输入 fit 的数据类型会给你带来一些麻烦] 或 fit_transform 方法不是 NumPy 数组。正如 Vivek 在他的评论中提到的，传递给 fit 方法的 X 不再是 DataFrame。不过我们还是先看看trace吧
ValueError: cannot label index with a null key
虽然 Vivek 对 NumPy 数组的看法是正确的，但您这里还有另一个问题。您得到的实际错误是 fit 方法中 column 的值是 None。如果您查看上面的 encoder 对象，您会看到 __repr__ 方法输出以下内容：
dummy_var_encoder(column_to_dummy=None)
使用 Pipeline 时，此参数会被初始化并传递给 GridSearchCV。这种行为也可以在交叉验证和搜索方法中看到，并且具有与输入参数不同名称的属性会导致此类问题。解决此问题将使您走上正确的道路。
这样修改 __init__ 方法将解决这个特定问题：
def __init__(self, column='default_col_name'):
self.column = column
print(self.column)
但是，一旦完成此操作，Vivek 提到的问题就会浮出水面，您将不得不处理它。这是我之前 运行 了解的内容，尽管不是专门针对 DataFrames。我在 Use sklearn GridSearchCV on custom class whose fit method takes 3 arguments 中想出了一个解决方案。基本上，我创建了一个实现 __getitem__ 方法的包装器，使数据的外观和行为方式能够通过 GridSearchCV、Pipeline 和其他方法中使用的验证方法交叉验证方法。
编辑
我进行了这些更改，看来您的问题来自验证方法 check_array。虽然使用 dtype=pd.DataFrame 调用此方法会起作用，但线性模型使用 dtype=np.float64 调用此方法会抛出错误。要解决这个问题，而不是将原始数据与你的虚拟数据连接起来，你可以 return 你的虚拟列并使用它们进行拟合。这是无论如何都应该做的事情，因为您不想在您尝试拟合的模型中同时包含虚拟列和原始数据。您也可以考虑 drop_first 选项，但我要跑题了。因此，像这样更改您的 fit 方法可以让整个过程按预期工作。
def transform(self, X_DF):
''' Update X_DF to have set of dummy-variables instead of orig column'''
# convert self-attribute to local var for ease of stepping through function
column = self.column
# add columns for new dummy vars, and drop original categorical column
dummy_matrix = pd.get_dummies(X_DF[column], prefix=column)
return dummy_matrix