1
我正在使用pyspark,我想從Mllib包執行線性迴歸。所以我想生成我自己的(大)數據來比較我的集羣性能與單節點python解釋器。如何將normalVectorRDD轉換爲Labeled Point數據?
from pyspark.mllib.random import RandomRDDs
u=RandomRDDs.normalVectorRDD(sc, 1000000000, 500)
u.take(5)
我得到這個:
array([ -1.13787491e+00, 3.68202613e-01, 9.59762136e-01,
6.33172122e-01, -1.91278957e+00, -1.17794680e+00,
-7.77179759e-01, -1.48368585e+00, 2.32369644e+00,...]
而且我想它解析爲LabeledPoint數據,因此它可以被LinearregressionwithSGD算法來識別。每行如下:
LabeledPoint(0.469112,[-0.282863,-1.509059,-1.135632,1.212112,-0.173215,0.119209,-1.044236,-0.861849,-2.104569,-0.494929,1.071804,0.721555,-0.706771,-1.039575,0.27186,-0.424972,0.56702,0.276232,-1.087401,-0.67369,0.113648,-1.478427,0.524988,0.404705])
第一個值作爲目標或標籤,其餘爲特徵。