2016-03-21 131 views
0

我是新來的Scala,需要幫助閱讀一段代碼。我現在正在尋找下一個代碼:Apache SparkALS.scala,並試圖理解它是如何工作的以及這個源代碼中涉及哪些類/對象。 目前我在代碼中遇到了line 166問題,因爲我無法理解SchemaUtils對象的性質。確定對象的類,Scala

我已經將源代碼複製到我的本地存儲庫,並得到SchemaUtils對象的識別錯誤。在我的pom.xml中,我定義了Spark版本1.6.1,但我認爲這個對象在這個版本中不再可用(可能是較老的版本)。因此Scala不承認。錯誤消息是:

not found: value SchemaUtils 

我該如何解決這個錯誤?

這裏是我的pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 

    <modelVersion>4.0.0</modelVersion> 
    <groupId>test.spark</groupId> 
    <artifactId>als</artifactId> 
    <version>0.0.1-SNAPSHOT</version> 

    <pluginRepositories> 
     <pluginRepository> 
      <id>scala-tools.org</id> 
      <name>Scala-tools Maven2 Repository</name> 
      <url>http://scala-tools.org/repo-releases</url> 
     </pluginRepository> 
    </pluginRepositories> 

    <dependencies> 

     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.11</artifactId> 
      <version>1.6.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-mllib_2.11</artifactId> 
      <version>1.6.1</version> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-sql_2.11</artifactId> 
      <version>1.6.1</version> 
     </dependency> 
     <!----> 
    </dependencies> 

    <build> 
     <plugins> 
      <!-- mixed scala/java compile --> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <executions> 
        <execution> 
         <id>compile</id> 
         <goals> 
          <goal>compile</goal> 
         </goals> 
         <phase>compile</phase> 
        </execution> 
        <execution> 
         <id>test-compile</id> 
         <goals> 
          <goal>testCompile</goal> 
         </goals> 
         <phase>test-compile</phase> 
        </execution> 
        <execution> 
         <phase>process-resources</phase> 
         <goals> 
          <goal>compile</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
      <plugin> 
       <artifactId>maven-compiler-plugin</artifactId> 
       <configuration> 
        <source>1.7</source> 
        <target>1.7</target> 
       </configuration> 
      </plugin> 
      <!-- for fatjar --> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-assembly-plugin</artifactId> 
       <version>2.4</version> 
       <configuration> 
        <descriptorRefs> 
         <descriptorRef>jar-with-dependencies</descriptorRef> 
        </descriptorRefs> 
       </configuration> 
       <executions> 
        <execution> 
         <id>assemble-all</id> 
         <phase>package</phase> 
         <goals> 
          <goal>single</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-jar-plugin</artifactId> 
       <configuration> 
        <archive> 
         <manifest> 
          <addClasspath>true</addClasspath> 
          <mainClass>fully.qualified.MainClass</mainClass> 
         </manifest> 
        </archive> 
       </configuration> 
      </plugin> 
     </plugins> 
     <pluginManagement> 
      <plugins> 
       <!--This plugin's configuration is used to store Eclipse m2e settings 
        only. It has no influence on the Maven build itself. --> 
       <plugin> 
        <groupId>org.eclipse.m2e</groupId> 
        <artifactId>lifecycle-mapping</artifactId> 
        <version>1.0.0</version> 
        <configuration> 
         <lifecycleMappingMetadata> 
          <pluginExecutions> 
           <pluginExecution> 
            <pluginExecutionFilter> 
             <groupId>org.scala-tools</groupId> 
             <artifactId> 
              maven-scala-plugin 
             </artifactId> 
             <versionRange> 
              [2.15.2,) 
             </versionRange> 
             <goals> 
              <goal>compile</goal> 
              <goal>testCompile</goal> 
             </goals> 
            </pluginExecutionFilter> 
            <action> 
             <execute></execute> 
            </action> 
           </pluginExecution> 
          </pluginExecutions> 
         </lifecycleMappingMetadata> 
        </configuration> 
       </plugin> 
      </plugins> 
     </pluginManagement> 
    </build> 
</project> 
+1

也許你應該發佈你的'pom.xml' – declension

+0

@Tzach Zohar,謝謝你的迴應。我希望,我已經做到了。如果你能幫助我,我發佈我的pom.xml以獲取詳細信息。 – Guforu

+0

由於它對於'org.apache.spark'是私有的,所以你不能在代碼中使用'SchemaUtils':https://github.com/apache/spark/blob/v1.4.0/mllib/src/main/scala/ org/apache/spark/ml/util/SchemaUtils.scala#L26 –

回答

1

Spark docs

對於Scala的API,星火1.6.1使用Scala的2.10。您將需要使用兼容的Scala版本(2.10.x)。

不能使用2.11星火(還),所以用編碼讀取_2.10和嘗試,斯卡拉版改爲<artifactId>spark-core_2.11</artifactId>和所有相關的依賴性。

+0

非常感謝。對不起,我真的是scala的新手 – Guforu

+0

沒問題,這是一種複雜的語言...... – declension

+0

1. Spark可以爲2.11構建。 http://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211 – Zernike