如何在SBT中使用測試罐作爲火花

Question

我正在創建Spark 2.0.1項目並希望在我的SBT項目中使用Spark測試罐。

build.sbt：

scalaVersion := "2.11.0" 
val sparkVersion = "2.0.1" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion % "compile", 
    "org.apache.spark" %% "spark-sql" % sparkVersion % "compile", 
    "org.scalatest" %% "scalatest" % "2.2.6" % "test", 
    "org.apache.spark" %% "spark-core" % sparkVersion % "test" classifier "tests", 
    "org.apache.spark" %% "spark-sql" % sparkVersion % "test" classifier "tests", 
    "org.apache.spark" %% "spark-catalyst" % sparkVersion % "test" classifier "tests" 
) 

我的測試代碼：

import org.apache.spark.sql.DataFrame 
import org.apache.spark.sql.functions._ 
import org.apache.spark.sql.test.SharedSQLContext 

class LoaderTest extends org.apache.spark.sql.QueryTest with SharedSQLContext { 
    import testImplicits._ 

    test("function current_date") { 
     val df1 = Seq((1, 2), (3, 1)).toDF("a", "b") 
     // Rest of test code and assertion using checkAnswer method 
    } 
} 

但是當我嘗試使用運行測試：

sbt clean test 

它得到以下錯誤：

[info] Compiling 1 Scala source to /tstprg/test/target/scala-2.11/test-classes... 
[error] bad symbolic reference to org.apache.spark.sql.catalyst.expressions.PredicateHelper encountered in class file 'PlanTest.class'. 
[error] Cannot access type PredicateHelper in package org.apache.spark.sql.catalyst.expressions. The current classpath may be 
[error] missing a definition for org.apache.spark.sql.catalyst.expressions.PredicateHelper, or PlanTest.class may have been compiled against a version that's 
[error] incompatible with the one found on the current classpath. 
[error] /tstprg/test/src/test/scala/facts/LoaderTest.scala:7: illegal inheritance; 
[error] self-type facts.LoaderTest does not conform to org.apache.spark.sql.QueryTest's selftype org.apache.spark.sql.QueryTest 
[error]  class LoaderTest extends org.apache.spark.sql.QueryTest with SharedSQLContext { 
[error]             ^
[error] /tstprg/test/src/test/scala/facts/LoaderTest.scala:7: illegal inheritance; 
[error] self-type facts.LoaderTest does not conform to org.apache.spark.sql.test.SharedSQLContext's selftype org.apache.spark.sql.test.SharedSQLContext 
[error]  class LoaderTest extends org.apache.spark.sql.QueryTest with SharedSQLContext { 
[error]                ^
[error] bad symbolic reference to org.apache.spark.sql.Encoder encountered in class file 'SQLImplicits.class'. 
[error] Cannot access type Encoder in package org.apache.spark.sql. The current classpath may be 
[error] missing a definition for org.apache.spark.sql.Encoder, or SQLImplicits.class may have been compiled against a version that's 
[error] incompatible with the one found on the current classpath. 
[error] /tstprg/test/src/test/scala/facts/LoaderTest.scala:11: bad symbolic reference to org.apache.spark.sql.catalyst.plans.logical encountered in class file 'SQLTestUtils.class'. 
[error] Cannot access term logical in package org.apache.spark.sql.catalyst.plans. The current classpath may be 
[error] missing a definition for org.apache.spark.sql.catalyst.plans.logical, or SQLTestUtils.class may have been compiled against a version that's 
[error] incompatible with the one found on the current classpath. 
[error]   val df1 = Seq((1, 2), (3, 1)).toDF("a", "b") 
[error]          ^
[error] 5 errors found 
[error] (test:compileIncremental) Compilation failed 

任何試過使用SBT進行單元測試的嘗試使用Spark的測試罐的人能幫助我瞭解什麼嗎？

注意：當我運行IntelliJ IDE時，此測試正常工作。

Answer 1

試着改變你的依賴標記爲測試的範圍像下面

scalaVersion := "2.11.0" 
val sparkVersion = "2.0.1" 

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion, 
    "org.apache.spark" %% "spark-sql" % sparkVersion, 
    "org.scalatest" %% "scalatest" % "2.2.6", 
    "org.apache.spark" %% "spark-core" % sparkVersion , 
    "org.apache.spark" %% "spark-sql" % sparkVersion, 
    "org.apache.spark" %% "spark-catalyst" % sparkVersion 
) 

或添加「編譯」。