如何打印預測的概率爲LogisticRegressionWithLBFGS pyspark

Question

我使用星火1.5.1和， 在pyspark，之後我適合使用的型號：

model = LogisticRegressionWithLBFGS.train(parsedData) 

我可以使用打印預測：

model.predict(p.features) 

是否有打印的概率得分也與預測沿功能？

Answer 1

你必須clear the threshold第一，這僅適用於二進制分類：

from pyspark.mllib.classification import LogisticRegressionWithLBFGS, LogisticRegressionModel 
from pyspark.mllib.regression import LabeledPoint 

parsed_data = [LabeledPoint(0.0, [4.6,3.6,1.0,0.2]), 
       LabeledPoint(0.0, [5.7,4.4,1.5,0.4]), 
       LabeledPoint(1.0, [6.7,3.1,4.4,1.4]), 
       LabeledPoint(0.0, [4.8,3.4,1.6,0.2]), 
       LabeledPoint(1.0, [4.4,3.2,1.3,0.2])] 

model = LogisticRegressionWithLBFGS.train(sc.parallelize(parsed_data)) 
model.threshold 
# 0.5 
model.predict(parsed_data[2].features) 
# 1 

model.clearThreshold() 
model.predict(parsed_data[2].features) 
# 0.9873840020002339 

Answer 2

我相信這個問題是在計算概率得分的預測整個訓練集。如果是這樣，我做了以下計算。不知道後仍然有效，但這是howI這樣做：

#get the original training data before it was converted to rows of LabelPoint. 
#let us assume it is otd (of type spark DataFrame) 
#let us extract the featureset as rdd by: 
fs=otd.rdd.map(lambda x:x[1:]) # assuming label is col 0. 

#the below is just a sample way of creating a Labelpoint rows.. 
parsedData= otd.rdd.map(lambda x: reg.LabeledPoint(int(x[0]-1),x[1:])) 

# now convert otd to a panda DataFrame as: 
ptd= otd.toPandas() 
m= ptd.shape[0] 
# train and get the model 
model=LogisticRegressionWithLBFGS.train(trainingData,numClasses=10) 


#Now store the model.predict rdd structures 
predict=model.predict(fs) 
pr=predict.collect() 

correct=0 
correct = ((ptd.label-1) == (pr)).sum() 
print((correct/m) *100) 

注意上面的是多級分類。