我在創建SparkSession時遇到IllegalArgumentException

Question

我在spark 2.1.0和python 2.7上使用pyspark和jupyter筆記本。我正在嘗試使用下面的代碼創建一個新的SparkSession;

from pyspark import SparkContext 
from pyspark import SparkConf 
from pyspark.sql import SparkSession 
from pyspark.sql import SQLContext 

spark = SparkSession\ 
    .builder\ 
    .appName("Bank Service Classifier")\ 
    .config("spark.sql.crossJoin.enabled","true")\ 
    .getOrCreate() 

sc = SparkContext() 
sqlContext = SQLContext(sc) 

但是，如果我得到以下錯誤;

IllegalArgumentException     Traceback (most recent call last) 
<ipython-input-40-2683a8d0ffcf> in <module>() 
     4 from pyspark.sql import SQLContext 
     5 
----> 6 spark = SparkSession .builder .appName("example-spark") .config("spark.sql.crossJoin.enabled","true") .getOrCreate() 
     7 
     8 sc = SparkContext() 

/srv/spark/python/pyspark/sql/session.py in getOrCreate(self) 
    177      session = SparkSession(sc) 
    178     for key, value in self._options.items(): 
--> 179      session._jsparkSession.sessionState().conf().setConfString(key, value) 
    180     for key, value in self._options.items(): 
    181      session.sparkContext._conf.set(key, value) 

/srv/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args) 
    1131   answer = self.gateway_client.send_command(command) 
    1132   return_value = get_return_value(
-> 1133    answer, self.gateway_client, self.target_id, self.name) 
    1134 
    1135   for temp_arg in temp_args: 

/srv/spark/python/pyspark/sql/utils.py in deco(*a, **kw) 
    77     raise QueryExecutionException(s.split(': ', 1)[1], stackTrace) 
    78    if s.startswith('java.lang.IllegalArgumentException: '): 
---> 79     raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace) 
    80    raise 
    81  return deco 

IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':" 

我該如何解決這個問題？

Answer 1

我遇到了這個相同的錯誤。下載Spark爲Hadoop 2.6預先構建而不是2.7爲我工作。