使用火花測試基的火花單元測試

Question

我在探討如何爲我的火花scala代碼編寫單元測試。我跨過火花測試基地，並在試圖運行一些標準的例子我得到一個錯誤來

import org.scalatest.FunSuite 
import com.holdenkarau.spark.testing.{SharedSparkContext} 

class AppTest extends FunSuite with SharedSparkContext { 
    test("test initializing spark context") { 
    val list = List(1, 2, 3, 4) 
    val rdd = sc.parallelize(list) 

    assert(rdd.count === list.length) 
    } 
} 

錯誤如下：

Class 'AppTest' must either be declared abstract or implement abstract member 'withFixture(test:Any):Outcome' in 'org.scalatest.SuiteMixin' 

我使用的IntelliJ IDEA的相同，我的build.sbt看起來如下

name := "" 

version := "1.0" 

scalaVersion := "2.11.8" 

fork in Test := true 

libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "2.0.0", 
    "org.apache.spark" %% "spark-mllib"    % "2.0.0", 
    "org.apache.spark" %% "spark-sql"    % "2.0.0", 
    "org.scalatest" %% "scalatest" % "1.9.1" % "test", 
    "com.holdenkarau" %% "spark-testing-base" % "0.7.2" % "test" 
) 

parallelExecution in Test := false 

我從來沒有做過單元測試。我將不勝感激這個問題的任何幫助

Answer 1

我建議不要使用火花測試基地。相反，您可以使用SparkContext的內置功能在運行單元測試過程中啓動。

例子：

import org.scalatest.FunSuite 

class AppTest extends FunSuite { 
    test("test initializing spark context") { 
    val myLocalInProcessSession = SparkSession 
            .builder 
            .master("local[*]") //This is the key config change to make it a local in process spark session. 
            .appName("myApp") 
            .getOrCreate() 
    val list = List(1, 2, 3, 4) 
    val rdd = myLocalInProcessSession.sparkContext.parallelize(list) 

    assert(rdd.count === list.length) 
    myLocalInProcessSession.close() 
    } 
} 

可以概括在FunSuite工具的本地火花測試環節的創建。