Python（boto）TypeError啓動Spark Cluster

以下是嘗試啓動具有10個從站的羣集。Python（boto）TypeError啓動Spark Cluster

12:13:44/sparkup $ec2/spark-ec2 -k sparkeast -i ~/.ssh/myPem.pem \ 
      -s 10 -z us-east-1a -r us-east-1 launch spark2

這裏是輸出。請注意，同一個命令在2月主代碼中已經成功。今天我已更新到最新的1.4.0-SNAPSHOT

Setting up security groups... 
Searching for existing cluster spark2 in region us-east-1... 
Spark AMI: ami-5bb18832 
Launching instances... 
Launched 10 slaves in us-east-1a, regid = r-68a0ae82 
Launched master in us-east-1a, regid = r-6ea0ae84 
Waiting for AWS to propagate instance metadata... 
Waiting for cluster to enter 'ssh-ready' state.........unable to load cexceptions 
TypeError 
p0 
(S'' 
p1 
tp2 
Rp3 
(dp4 
S'child_traceback' 
p5 
S'Traceback (most recent call last):\n File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1280, in _execute_child\n sys.stderr.write("%s %s (env=%s)\\n" %(executable, \' \'.join(args), \' \'.join(env)))\nTypeError\n' 
p6 
sb.Traceback (most recent call last): 
    File "ec2/spark_ec2.py", line 1444, in <module> 
    main() 
    File "ec2/spark_ec2.py", line 1436, in main 
    real_main() 
    File "ec2/spark_ec2.py", line 1270, in real_main 
    cluster_state='ssh-ready' 
    File "ec2/spark_ec2.py", line 869, in wait_for_cluster_state 
    is_cluster_ssh_available(cluster_instances, opts): 
    File "ec2/spark_ec2.py", line 833, in is_cluster_ssh_available 
    if not is_ssh_available(host=dns_name, opts=opts): 
    File "ec2/spark_ec2.py", line 807, in is_ssh_available 
    stderr=subprocess.STDOUT # we pipe stderr through stdout to preserve output order 
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in __init__ 
    errread, errwrite) 
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1328, in _execute_child 
    raise child_exception 
TypeError

AWS控制檯顯示實際上正在運行實例。所以目前還不清楚實際失敗的原因。

enter image description here 讚賞任何提示或解決方法。

UPDATE執行登錄命令時會發生同樣的錯誤。這似乎是與博託API問題 - 但集羣本身似乎是好的。

ec2/spark-ec2 -i ~/.ssh/sparkeast.pem login spark2 
Searching for existing cluster spark2 in region us-east-1... 
Found 1 master, 10 slaves. 
Logging into master ec2-54-87-46-170.compute-1.amazonaws.com... 
unable to load cexceptions 
TypeError 
p0 
(.. same exception stacktrace as above)

來源

2015-05-09 javadba

我對你使用的''spark_ec2''模塊一無所知，但是從堆棧跟蹤看來，它在發生錯誤時似乎沒有做與boto相關的任何事情。它看起來像是在等待SSH響應實例。 – garnaat

@garnaat - 是的，看起來更像是調用'ssh'的'subprocess'調用出了問題。 javadba - 你的'ssh'可執行嗎？ –

re：ssh我有其他的ssh窗口每天打開並打開/關閉ssh窗口多次。我也曾多次與AWS連接，最近使用的是2月的Spark代碼庫。可以肯定的是，我只是向合作伙伴的羣集打開ssh，並沒有問題。 – javadba

問題是我的優勝美地macbook上的python-2.7.6安裝似乎已損壞。

我重置PATH和PYTHONPATH指向一個自定義的自制軟件安裝的蟒蛇版本，然後博託 - 和其他python命令，包括建設火花性能項目 - 工作正常。

來源

2015-05-10 06:57:32 javadba

Python（boto）TypeError啓動Spark Cluster

回答

相關問題