VertexRDD給我類型不匹配錯誤

Question

我正在運行下面的代碼，試圖在Apache Spark中的GraphX中創建圖。

import org.apache.spark.SparkConf 

import org.apache.spark.SparkContext 

import org.apache.spark.graphx.GraphLoader 

import org.apache.spark.graphx.Graph 

import org.apache.spark.rdd.RDD 
import org.apache.spark.graphx.VertexId 

//loads file from the array 

val lines = sc.textFile("hdfs://moonshot-ha-nameservice/data/google-plus/2309.graph"); 

//maps lines and takes the first 21 characters of each line which is the node. 

val result = lines.map(line => line.substring(0,20)) 

//creates a new variable with each node followed by a long . 

val result2 = result.map(word => (word,1L).toLong) 

//where i am getting an error 

val vertexRDD: RDD[(Long,Long)] = sc.parallelize(result2) 

我收到以下錯誤：

error: type mismatch; 

found : org.apache.spark.rdd.RDD[(Long, Long)] 

required: Seq[?] 

Error occurred in an application involving default arguments. 
     val vertexRDD: RDD[(Long, Long)] = sc.parallelize(result2) 

Answer 1

首先，您的地圖可以簡化爲以下代碼：現在

val vertexRDD: RDD[(Long, Long)] = 
    lines.map(line => (line.substring(0, 17).toLong, 1L)) 

，你的錯誤：你不能用RDD撥打sc.parallelize。您的vertexRDD已經由result2定義。然後，您可以創建RESULT2您的圖形和你EdgesRDD：

val g = Graph(result2, edgesRDD) 

，或者，如果採用我的建議：

val g = Graph(vertexRDD, edgesRDD)