使用 SVM 预测概率
Predict probabilities using SVM
我写了这段代码，想获得class化的概率。
from sklearn import svm
X = [[0, 0], [10, 10],[20,30],[30,30],[40, 30], [80,60], [80,50]]
y = [0, 1, 2, 3, 4, 5, 6]
clf = svm.SVC()
clf.probability=True
clf.fit(X, y)
prob = clf.predict_proba([[10, 10]])
print prob
我得到了这个输出：
[[0.15376986 0.07691205 0.15388546 0.15389275 0.15386348 0.15383004 0.15384636]]
这很奇怪，因为概率应该是
[0 1 0 0 0 0 0 0]
（观察必须预测 class 的样本与第 2 个样本相同）同样，class 获得的概率是最低的。
您应该禁用 probability 并改用 decision_function，因为无法保证 predict_proba 和 predict return 的结果相同。
您可以在 documentation.
中阅读更多相关信息
clf.predict([[10, 10]]) // returns 1 as expected
prop = clf.decision_function([[10, 10]]) // returns [[ 4.91666667 6.5 3.91666667 2.91666667 1.91666667 0.91666667
-0.08333333]]
prediction = np.argmax(prop) // returns 1
你可以read in the docs那...
The SVC method decision_function gives per-class scores for each sample (or a single score per sample in the binary case). When the constructor option probability is set to True, class membership probability estimates (from the methods predict_proba and predict_log_proba) are enabled. In the binary case, the probabilities are calibrated using Platt scaling: logistic regression on the SVM’s scores, fit by an additional cross-validation on the training data. In the multiclass case, this is extended as per Wu et al. (2004).
Needless to say, the cross-validation involved in Platt scaling is an expensive operation for large datasets. In addition, the probability estimates may be inconsistent with the scores, in the sense that the “argmax” of the scores may not be the argmax of the probabilities. (E.g., in binary classification, a sample may be labeled by predict as belonging to a class that has probability <½ according to predict_proba.) Platt’s method is also known to have theoretical issues. If confidence scores are required, but these do not have to be probabilities, then it is advisable to set probability=False and use decision_function instead of predict_proba.
Stack Overflow 用户对此函数也有很多困惑，如您在 this thread, or this one 中所见。
编辑：正如@TimH 所指出的，概率可以由clf.decision_function(X) 给出。下面的代码是固定的。注意到使用 predict_proba(X) 的低概率指定问题，我认为答案是根据官方文档 here、...此外，它会在非常小的数据集上产生无意义的结果.
答案在于理解 SVM 的结果概率是多少。
简而言之，您在 2D 平面中有 7 classes 和 7 个点。
SVM 试图做的是在每个 class 和每个 one-vs-one 之间找到一个线性分隔符（one-vs-one 方法）。每次只选择2classes。
你得到的是 class 投票者的投票，经过标准化 。请参阅 this post or here（scikit-learn 使用 libsvm）中关于 libsvm 的 multi-class SVM 的更详细解释。
通过稍微修改您的代码，我们看到确实选择了正确的 class：
from sklearn import svm
import matplotlib.pyplot as plt
import numpy as np
X = [[0, 0], [10, 10],[20,30],[30,30],[40, 30], [80,60], [80,50]]
y = [0, 1, 2, 3, 3, 4, 4]
clf = svm.SVC()
clf.fit(X, y)
x_pred = [[10,10]]
p = np.array(clf.decision_function(x_pred)) # decision is a voting function
prob = np.exp(p)/np.sum(np.exp(p),axis=1, keepdims=True) # softmax after the voting
classes = clf.predict(x_pred)
_ = [print('Sample={}, Prediction={},\n Votes={} \nP={}, '.format(idx,c,v, s)) for idx, (v,s,c) in enumerate(zip(p,prob,classes))]
对应的输出为
Sample=0, Prediction=0,
Votes=[ 6.5 4.91666667 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333]
P=[ 0.75531071 0.15505748 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477],
Sample=1, Prediction=1,
Votes=[ 4.91666667 6.5 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333]
P=[ 0.15505748 0.75531071 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477],
Sample=2, Prediction=2,
Votes=[ 1.91666667 2.91666667 6.5 4.91666667 3.91666667 0.91666667 -0.08333333]
P=[ 0.00771986 0.02098475 0.75531071 0.15505748 0.05704246 0.00283998 0.00104477],
Sample=3, Prediction=3,
Votes=[ 1.91666667 2.91666667 4.91666667 6.5 3.91666667 0.91666667 -0.08333333]
P=[ 0.00771986 0.02098475 0.15505748 0.75531071 0.05704246 0.00283998 0.00104477],
Sample=4, Prediction=4,
Votes=[ 1.91666667 2.91666667 3.91666667 4.91666667 6.5 0.91666667 -0.08333333]
P=[ 0.00771986 0.02098475 0.05704246 0.15505748 0.75531071 0.00283998 0.00104477],
Sample=5, Prediction=5,
Votes=[ 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333 6.5 4.91666667]
P=[ 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477 0.75531071 0.15505748],
Sample=6, Prediction=6,
Votes=[ 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333 4.91666667 6.5 ]
P=[ 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477 0.15505748 0.75531071],
你还可以看到决策区：
X = np.array(X)
y = np.array(y)
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111)
XX, YY = np.mgrid[0:100:200j, 0:100:200j]
Z = clf.predict(np.c_[XX.ravel(), YY.ravel()])
Z = Z.reshape(XX.shape)
plt.figure(1, figsize=(4, 3))
plt.pcolormesh(XX, YY, Z, cmap=plt.cm.Paired)
for idx in range(7):
ax.scatter(X[idx,0],X[idx,1], color='k')
我写了这段代码，想获得class化的概率。
from sklearn import svm
X = [[0, 0], [10, 10],[20,30],[30,30],[40, 30], [80,60], [80,50]]
y = [0, 1, 2, 3, 4, 5, 6]
clf = svm.SVC()
clf.probability=True
clf.fit(X, y)
prob = clf.predict_proba([[10, 10]])
print prob
我得到了这个输出：
[[0.15376986 0.07691205 0.15388546 0.15389275 0.15386348 0.15383004 0.15384636]]
这很奇怪，因为概率应该是
[0 1 0 0 0 0 0 0]
（观察必须预测 class 的样本与第 2 个样本相同）同样，class 获得的概率是最低的。
您应该禁用 probability 并改用 decision_function，因为无法保证 predict_proba 和 predict return 的结果相同。
您可以在 documentation.
clf.predict([[10, 10]]) // returns 1 as expected
prop = clf.decision_function([[10, 10]]) // returns [[ 4.91666667 6.5 3.91666667 2.91666667 1.91666667 0.91666667
-0.08333333]]
prediction = np.argmax(prop) // returns 1
你可以read in the docs那...
The SVC method decision_function gives per-class scores for each sample (or a single score per sample in the binary case). When the constructor option probability is set to True, class membership probability estimates (from the methods predict_proba and predict_log_proba) are enabled. In the binary case, the probabilities are calibrated using Platt scaling: logistic regression on the SVM’s scores, fit by an additional cross-validation on the training data. In the multiclass case, this is extended as per Wu et al. (2004).
Needless to say, the cross-validation involved in Platt scaling is an expensive operation for large datasets. In addition, the probability estimates may be inconsistent with the scores, in the sense that the “argmax” of the scores may not be the argmax of the probabilities. (E.g., in binary classification, a sample may be labeled by predict as belonging to a class that has probability <½ according to predict_proba.) Platt’s method is also known to have theoretical issues. If confidence scores are required, but these do not have to be probabilities, then it is advisable to set probability=False and use decision_function instead of predict_proba.
Stack Overflow 用户对此函数也有很多困惑，如您在 this thread, or this one 中所见。
编辑：正如@TimH 所指出的，概率可以由clf.decision_function(X) 给出。下面的代码是固定的。注意到使用 predict_proba(X) 的低概率指定问题，我认为答案是根据官方文档 here、...此外，它会在非常小的数据集上产生无意义的结果.
答案在于理解 SVM 的结果概率是多少。 简而言之，您在 2D 平面中有 7 classes 和 7 个点。 SVM 试图做的是在每个 class 和每个 one-vs-one 之间找到一个线性分隔符（one-vs-one 方法）。每次只选择2classes。 你得到的是 class 投票者的投票，经过标准化 。请参阅 this post or here（scikit-learn 使用 libsvm）中关于 libsvm 的 multi-class SVM 的更详细解释。
通过稍微修改您的代码，我们看到确实选择了正确的 class：
from sklearn import svm
import matplotlib.pyplot as plt
import numpy as np
X = [[0, 0], [10, 10],[20,30],[30,30],[40, 30], [80,60], [80,50]]
y = [0, 1, 2, 3, 3, 4, 4]
clf = svm.SVC()
clf.fit(X, y)
x_pred = [[10,10]]
p = np.array(clf.decision_function(x_pred)) # decision is a voting function
prob = np.exp(p)/np.sum(np.exp(p),axis=1, keepdims=True) # softmax after the voting
classes = clf.predict(x_pred)
_ = [print('Sample={}, Prediction={},\n Votes={} \nP={}, '.format(idx,c,v, s)) for idx, (v,s,c) in enumerate(zip(p,prob,classes))]
对应的输出为
Sample=0, Prediction=0,
Votes=[ 6.5 4.91666667 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333]
P=[ 0.75531071 0.15505748 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477],
Sample=1, Prediction=1,
Votes=[ 4.91666667 6.5 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333]
P=[ 0.15505748 0.75531071 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477],
Sample=2, Prediction=2,
Votes=[ 1.91666667 2.91666667 6.5 4.91666667 3.91666667 0.91666667 -0.08333333]
P=[ 0.00771986 0.02098475 0.75531071 0.15505748 0.05704246 0.00283998 0.00104477],
Sample=3, Prediction=3,
Votes=[ 1.91666667 2.91666667 4.91666667 6.5 3.91666667 0.91666667 -0.08333333]
P=[ 0.00771986 0.02098475 0.15505748 0.75531071 0.05704246 0.00283998 0.00104477],
Sample=4, Prediction=4,
Votes=[ 1.91666667 2.91666667 3.91666667 4.91666667 6.5 0.91666667 -0.08333333]
P=[ 0.00771986 0.02098475 0.05704246 0.15505748 0.75531071 0.00283998 0.00104477],
Sample=5, Prediction=5,
Votes=[ 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333 6.5 4.91666667]
P=[ 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477 0.75531071 0.15505748],
Sample=6, Prediction=6,
Votes=[ 3.91666667 2.91666667 1.91666667 0.91666667 -0.08333333 4.91666667 6.5 ]
P=[ 0.05704246 0.02098475 0.00771986 0.00283998 0.00104477 0.15505748 0.75531071],
你还可以看到决策区：
X = np.array(X)
y = np.array(y)
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111)
XX, YY = np.mgrid[0:100:200j, 0:100:200j]
Z = clf.predict(np.c_[XX.ravel(), YY.ravel()])
Z = Z.reshape(XX.shape)
plt.figure(1, figsize=(4, 3))
plt.pcolormesh(XX, YY, Z, cmap=plt.cm.Paired)
for idx in range(7):
ax.scatter(X[idx,0],X[idx,1], color='k')