java apache spark mllib

Question

我已經開始在java中學習mllib apache spark了。我從官方網站跟隨spark 2.1.1文檔。我在我的ubuntu 14.04 lts中安裝了spark-2.1.1-bin-hadoop2.7。我正在嘗試運行此代碼。

public class JavaLogisticRegressionWithElasticNetExample { 
public static void main(String[] args) { 
    SparkSession spark = SparkSession.builder().appName("JavaLogisticRegressionWithElasticNetExample") .master("local[*]").getOrCreate(); 
    // $example on$ 
    // Load training data 
    Dataset<Row> training = spark.read().format("libsvm") 
      .load("data/mllib/sample_libsvm_data.txt"); 

    LogisticRegression lr = new LogisticRegression() 
      .setMaxIter(10) 
      .setRegParam(0.3) 
      .setElasticNetParam(0.8); 

    // Fit the model 
    LogisticRegressionModel lrModel = lr.fit(training); 

    // Print the coefficients and intercept for logistic regression 
    System.out.println("Coefficients: " 
      + lrModel.coefficients() + " Intercept: " + lrModel.intercept()); 

    // We can also use the multinomial family for binary classification 
    LogisticRegression mlr = new LogisticRegression() 
      .setMaxIter(10) 
      .setRegParam(0.3) 
      .setElasticNetParam(0.8) 
      .setFamily("multinomial"); 

    // Fit the model 
    LogisticRegressionModel mlrModel = mlr.fit(training); 

    // Print the coefficients and intercepts for logistic regression with multinomial family 
    System.out.println("Multinomial coefficients: " + lrModel.coefficientMatrix() 
      + "\nMultinomial intercepts: " + mlrModel.interceptVector()); 
    // $example off$ 

    spark.stop(); 
} 

}

我在我的系統中安裝了火花2.1.1彬hadoop2.7。 我的pom.xml文件

<dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.11</artifactId> 
     <version>2.1.1</version> 
     <scope>provided</scope> 
    </dependency> 
<dependency> 
    <groupId>org.apache.spark</groupId> 
    <artifactId>spark-mllib_2.10</artifactId> 
    <version>2.1.1</version> 
</dependency> 
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib-local_2.10 --> 
<dependency> 
    <groupId>org.apache.spark</groupId> 
    <artifactId>spark-mllib-local_2.10</artifactId> 
    <version>2.1.1</version> 
</dependency> 

，但我得到這個例外

17/09/08 16:42:19 INFO SparkEnv: Registering OutputCommitCoordinator Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$scope()Lscala/xml/TopScope$; at org.apache.spark.ui.jobs.AllJobsPage.(AllJobsPage.scala:39) at org.apache.spark.ui.jobs.JobsTab.(JobsTab.scala:38) at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:65) at org.apache.spark.ui.SparkUI.(SparkUI.scala:82) at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:220) at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:162) at org.apache.spark.SparkContext.(SparkContext.scala:452) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2320) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at JavaLogisticRegressionWithElasticNetExample.main(JavaLogisticRegressionWithElasticNetExample.java:12) 17/09/08 16:42:19 INFO DiskBlockManager: Shutdown hook called 17/09/08 16:42:19 INFO ShutdownHookManager: Shutdown hook called 17/09/08 16:42:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-8460a189-3039-47ec-8d75-9e0ca8b4ee5d 17/09/08 16:42:19 INFO ShutdownHookManager: Deleting directory /tmp/spark-8460a189-3039-47ec-8d75-9e0ca8b4ee5d/userFiles-9b6994eb-1376-47a3-929e-e415e1fdb0c0

Answer 1

當您使用不同版本的階同一程序中的這種錯誤的發生。事實上，在你的依賴關係中（你的pom.xml），你有一些scala 2.10的庫和scala 2.11的庫。

使用spark-sql_2.10而不是spark-sql_2.11，你會沒事的（或將mllib版本更改爲2.11）。