IPython的筆記本火花得到誤差sparkcontext

我用這個例子的測試圖裏對我的MacBook OSX 10.10.5 https://turi.com/learn/gallery/notebooks/spark_and_graphlab_create.html IPython的筆記本火花得到誤差sparkcontext

得到這一步

# Set up the SparkContext object 
# this can be 'local' or 'yarn-client' in PySpark 
# Remember if using yarn-client then all the paths should be accessible 
# by all nodes in the cluster. 
sc = SparkContext('local')

以下錯誤，當出現

--------------------------------------------------------------------------- 
Exception         Traceback (most recent call last) 
<ipython-input-12-dc1befb4186c> in <module>() 
     3 # Remember if using yarn-client then all the paths should be accessible 
     4 # by all nodes in the cluster. 
----> 5 sc = SparkContext() 

/usr/local/Cellar/apache-spark/1.6.2/libexec/python/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls) 
    110   """ 
    111   self._callsite = first_spark_call() or CallSite(None, None, None) 
--> 112   SparkContext._ensure_initialized(self, gateway=gateway) 
    113   try: 
    114    self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer, 

/usr/local/Cellar/apache-spark/1.6.2/libexec/python/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway) 
    243   with SparkContext._lock: 
    244    if not SparkContext._gateway: 
--> 245     SparkContext._gateway = gateway or launch_gateway() 
    246     SparkContext._jvm = SparkContext._gateway.jvm 
    247 

/usr/local/Cellar/apache-spark/1.6.2/libexec/python/pyspark/java_gateway.pyc in launch_gateway() 
    92     callback_socket.close() 
    93   if gateway_port is None: 
---> 94    raise Exception("Java gateway process exited before sending the driver its port number") 
    95 
    96   # In Windows, ensure the Java child processes do not linger after Python has exited. 

Exception: Java gateway process exited before sending the driver its port number

快速谷歌搜索沒有幫助。

，這裏是我的.bash_profile

# added by Anaconda2 4.1.1 installer 
export PATH="/Users/me/anaconda/bin:$PATH" 

export SCALA_HOME=/usr/local/Cellar/scala/2.11.8/libexec 
export SPARK_HOME=/usr/local/Cellar/apache-spark/1.6.2/libexec 
export PYTHONPATH=$SPARK_HOME/python/pyspark:$PYTHONPATH 
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH 
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH

任何人都知道如何解決這個問題？

感謝

來源

2016-07-13 ikel

「SPARK_HOME」路徑是否正確？您是否在環境變量中設置了「PYSPARK_SUBMIT_ARGS =」 - master spark：// ：「'？這可能是你缺少的端口號 – KartikKannapur

spark_home是正確的，我還沒有配置PYSPARK_SUBMIT_ARGS，在這種情況下我應該指定什麼？ – ikel

嘗試這個'出口PYSPARK_SUBMIT_ARGS =「 - 本地主[2]」' – KartikKannapur

這可能會發生的原因有兩個：

環境變量SPARK_HOME可能指向錯誤的路徑
設置export PYSPARK_SUBMIT_ARGS="--master local[2]" - 這是配置你想PySpark入手。

來源

2016-07-13 15:13:30 KartikKannapur

IPython的筆記本火花得到誤差sparkcontext

回答

相關問題