2017-07-16 41 views
1

我正在使用pyspark,我想從Mllib包執行線性迴歸。所以我想生成我自己的(大)數據來比較我的集羣性能與單節點python解釋器。如何將normalVectorRDD轉換爲Labeled Point數據?

from pyspark.mllib.random import RandomRDDs 
u=RandomRDDs.normalVectorRDD(sc, 1000000000, 500) 
u.take(5) 

我得到這個:

array([ -1.13787491e+00, 3.68202613e-01, 9.59762136e-01, 
     6.33172122e-01, -1.91278957e+00, -1.17794680e+00, 
    -7.77179759e-01, -1.48368585e+00, 2.32369644e+00,...] 

而且我想它解析爲LabeledPoint數據,因此它可以被LinearregressionwithSGD算法來識別。每行如下:

LabeledPoint(0.469112,[-0.282863,-1.509059,-1.135632,1.212112,-0.173215,0.119209,-1.044236,-0.861849,-2.104569,-0.494929,1.071804,0.721555,-0.706771,-1.039575,0.27186,-0.424972,0.56702,0.276232,-1.087401,-0.67369,0.113648,-1.478427,0.524988,0.404705]) 

第一個值作爲目標或標籤,其餘爲特徵。

回答

1

試試這個,

from pyspark.mllib.regression import LabeledPoint 
u.map(lambda x:LabeledPoint(x[0],x[1:]))