隨機森林使用pyspark.ml爲Dataframes

Question

我想建立使用dataframes pyspark.ml庫（不mllib爲RDD）隨機森林分類。 我是否必須使用文檔中給出的管道？ 我只是想建立一個簡單的模型，

rf = RandomForestClassifier(labelCol = labs, featuresCol = rawdata) 

我碰到下面的錯誤

Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/lib/spark/python/pyspark/__init__.py", line 104, in wrapper 
    return func(self, **kwargs) 
    File "/usr/lib/spark/python/pyspark/ml/classification.py", line 910, in __init 
__ 
    self.setParams(**kwargs) 
    File "/usr/lib/spark/python/pyspark/__init__.py", line 104, in wrapper 
    return func(self, **kwargs) 
    File "/usr/lib/spark/python/pyspark/ml/classification.py", line 928, in setPar 
ams 
    return self._set(**kwargs) 
    File "/usr/lib/spark/python/pyspark/ml/param/__init__.py", line 421, in _set 
    raise TypeError('Invalid param value given for param "%s". %s' % (p.name, e) 
) 
TypeError: Invalid param value given for param "labelCol". Could not convert <cl 
ass 'pyspark.sql.dataframe.DataFrame'> to string type 

我的標籤樣品

+---+ 
| _2| 
+---+ 
|0.0| 
|1.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|1.0| 
|1.0| 
|1.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|0.0| 
|1.0| 
|1.0| 
+---+ 

我的數據是180列類似。

Answer 1

Spark Spark數據框不像Spark ML中那樣使用;您的所有功能都必須是單個列中的矢量，通常（但不一定）將其命名爲features。另外，labelcol=labs表示您的標籤必須位於名爲labs的列中，而不是_2。

下面是一個例子，有想法，有玩具數據：

spark.version 
# u'2.2.0' 

from pyspark.ml.classification import RandomForestClassifier 
from pyspark.ml.linalg import Vectors 
df = sqlContext.createDataFrame([ 
    (0.0, Vectors.dense(0.0, 1.0)), 
    (1.0, Vectors.dense(1.0, 0.0))], 
    ["label", "features"]) 

df.show() # notice there are only 2 columns, and 'features' is a 2-d vector 
# +-----+---------+ 
# |label| features| 
# +-----+---------+ 
# | 0.0|[0.0,1.0]| 
# | 1.0|[1.0,0.0]| 
# +-----+---------+ 

rf = RandomForestClassifier(numTrees=3, maxDepth=2, labelCol="label", seed=42) 
rf_model = rf.fit(df) 

This answer of mine可能是如何將數據轉換成所需的格式幫助。