運行PySparkling與Spark 2.1的H20Context出錯

嘗試在AWS EMR集羣上運行Pysparkling腳本時出現此錯誤。當下載Sparkling water 2.1.8並從pysparkling shell運行它時，我可以讓所有的東西都能正常工作。但是，spark-submit似乎不起作用。運行PySparkling與Spark 2.1的H20Context出錯

錯誤：

NameError: name 'H2OContext' is not defined

我的火花提交：

spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-water-examples_2.11:2.1.7 --conf spark.dynamicAllocation.enabled=false spark.py

Python的文件

from pysparkling import * 

hc = H2OContext.getOrCreate(sc)

而且，我其實是試圖使火花背景，但只是導致同樣的錯誤，但需要更長的時間。

引導文件：

#!/usr/bin/env bash 


# install conda (conda 4.2 defaults to pythone35) 
wget --quiet http://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh -O ~/anaconda.sh \ 
    && /bin/bash ~/anaconda.sh -b -p $HOME/conda 

echo -e '\nexport PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc 

# install packages 
conda install -y ipython jupyter 

# needed for PySparkling 
conda install requests 
conda install six 
conda install future 
conda install tabulate 

# install pysparkling 
pip install h2o 
# pip install pysparkling 
pip install h2o_pysparkling_2.1

更詳細的輸出：

[[email protected] test]$ spark-submit --packages ai.h2o:sparkling-water-core_2.11:2.1.7,ai.h2o:sparkling-water-examples_2.11:2.1.7 --conf spark.dynamicAllocation.enabled=false spark.py 
Ivy Default Cache set to: /home/hadoop/.ivy2/cache 
The jars for the packages stored in: /home/hadoop/.ivy2/jars 
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml 
ai.h2o#sparkling-water-core_2.11 added as a dependency 
ai.h2o#sparkling-water-examples_2.11 added as a dependency 
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 
     confs: [default] 
     found ai.h2o#sparkling-water-core_2.11;2.1.7 in central 
     found ai.h2o#h2o-genmodel;3.10.4.7 in central 
     found net.sf.opencsv#opencsv;2.3 in central 
     found ai.h2o#deepwater-backend-api;1.0.2 in central 
     found com.google.guava#guava;19.0 in central 
     found ai.h2o#h2o-core;3.10.4.7 in central 
     found joda-time#joda-time;2.3 in central 
     found gov.nist.math#jama;1.0.3 in central 
     found org.javassist#javassist;3.18.2-GA in central 
     found org.apache.commons#commons-math3;3.3 in central 
     found commons-io#commons-io;2.4 in central 
     found ai.h2o#google-analytics-java;1.1.2-H2O-CUSTOM in central 
     found org.apache.httpcomponents#httpclient;4.1 in central 
     found org.apache.httpcomponents#httpcore;4.1 in central 
     found commons-logging#commons-logging;1.1.1 in central 
     found commons-codec#commons-codec;1.4 in central 
     found org.eclipse.jetty.aggregate#jetty-servlet;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-plus;8.1.17.v20150415 in central 
     found org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 in central 
     found org.eclipse.jetty#jetty-webapp;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-xml;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-util;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-servlet;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-security;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-server;8.1.17.v20150415 in central 
     found org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 in central 
     found org.eclipse.jetty#jetty-continuation;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-http;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-io;8.1.17.v20150415 in central 
     found org.eclipse.jetty#jetty-jndi;8.1.17.v20150415 in central 
     found org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020 in central 
     found org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 in central 
     found com.github.rwl#jtransforms;2.4.0 in central 
     found ai.h2o#h2o-jaas-pam;3.10.4.7 in central 
     found org.kohsuke#libpam4j;1.8 in central 
     found net.java.dev.jna#jna;4.0.0 in central 
     found log4j#log4j;1.2.15 in central 
     found com.google.code.gson#gson;2.3.1 in central 
     found commons-lang#commons-lang;2.6 in central 
     found ai.h2o#reflections;0.9.11-h2o-custom in central 
     found com.google.code.findbugs#jsr305;3.0.0 in central 
     found ai.h2o#h2o-algos;3.10.4.7 in central 
     found ai.h2o#h2o-web;3.10.4.7 in central 
     found ai.h2o#h2o-avro-parser;3.10.4.7 in central 
     found ai.h2o#h2o-parquet-parser;3.10.4.7 in central 
     found ai.h2o#h2o-orc-parser;3.10.4.7 in central 
     found ai.h2o#h2o-scala_2.11;3.10.4.7 in central 
     found ai.h2o#h2o-persist-hdfs;3.10.4.7 in central 
     found net.java.dev.jets3t#jets3t;0.6.1 in central 
     found commons-httpclient#commons-httpclient;3.1 in central 
     found ai.h2o#h2o-persist-s3;3.10.4.7 in central 
     found com.amazonaws#aws-java-sdk-s3;1.10.47 in central 
     found com.amazonaws#aws-java-sdk-kms;1.10.47 in central 
     found com.amazonaws#aws-java-sdk-core;1.10.47 in central 
     found commons-logging#commons-logging;1.1.3 in central 
     found org.apache.httpcomponents#httpclient;4.3.6 in central 
     found org.apache.httpcomponents#httpcore;4.3.3 in central 
     found commons-codec#commons-codec;1.6 in central 
     found joda-time#joda-time;2.8.1 in central 
     found ai.h2o#sparkling-water-repl_2.11;2.1.7 in central 
     found org.joda#joda-convert;1.7 in central 
     found ai.h2o#sparkling-water-examples_2.11;2.1.7 in central 
     found ai.h2o#sparkling-water-ml_2.11;2.1.7 in central 
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-core_2.11/2.1.7/sparkling-water-core_2.11-2.1.7.jar ... 
     [SUCCESSFUL ] ai.h2o#sparkling-water-core_2.11;2.1.7!sparkling-water-core_2.11.jar (56ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-examples_2.11/2.1.7/sparkling-water-examples_2.11-2.1.7.jar ... 
     [SUCCESSFUL ] ai.h2o#sparkling-water-examples_2.11;2.1.7!sparkling-water-examples_2.11.jar (15ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-genmodel/3.10.4.7/h2o-genmodel-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-genmodel;3.10.4.7!h2o-genmodel.jar (7ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-core/3.10.4.7/h2o-core-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-core;3.10.4.7!h2o-core.jar (129ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-algos/3.10.4.7/h2o-algos-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-algos;3.10.4.7!h2o-algos.jar (35ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-web/3.10.4.7/h2o-web-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-web;3.10.4.7!h2o-web.jar (512ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-scala_2.11/3.10.4.7/h2o-scala_2.11-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-scala_2.11;3.10.4.7!h2o-scala_2.11.jar (4ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-persist-hdfs/3.10.4.7/h2o-persist-hdfs-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-persist-hdfs;3.10.4.7!h2o-persist-hdfs.jar (2ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-persist-s3/3.10.4.7/h2o-persist-s3-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-persist-s3;3.10.4.7!h2o-persist-s3.jar (2ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-repl_2.11/2.1.7/sparkling-water-repl_2.11-2.1.7.jar ... 
     [SUCCESSFUL ] ai.h2o#sparkling-water-repl_2.11;2.1.7!sparkling-water-repl_2.11.jar (4ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/h2o-jaas-pam/3.10.4.7/h2o-jaas-pam-3.10.4.7.jar ... 
     [SUCCESSFUL ] ai.h2o#h2o-jaas-pam;3.10.4.7!h2o-jaas-pam.jar (2ms) 
downloading https://repo1.maven.org/maven2/ai/h2o/sparkling-water-ml_2.11/2.1.7/sparkling-water-ml_2.11-2.1.7.jar ... 
     [SUCCESSFUL ] ai.h2o#sparkling-water-ml_2.11;2.1.7!sparkling-water-ml_2.11.jar (10ms) 
:: resolution report :: resolve 6024ms :: artifacts dl 802ms 
     :: modules in use: 
     ai.h2o#deepwater-backend-api;1.0.2 from central in [default] 
     ai.h2o#google-analytics-java;1.1.2-H2O-CUSTOM from central in [default] 
     ai.h2o#h2o-algos;3.10.4.7 from central in [default] 
     ai.h2o#h2o-avro-parser;3.10.4.7 from central in [default] 
     ai.h2o#h2o-core;3.10.4.7 from central in [default] 
     ai.h2o#h2o-genmodel;3.10.4.7 from central in [default] 
     ai.h2o#h2o-jaas-pam;3.10.4.7 from central in [default] 
     ai.h2o#h2o-orc-parser;3.10.4.7 from central in [default] 
     ai.h2o#h2o-parquet-parser;3.10.4.7 from central in [default] 
     ai.h2o#h2o-persist-hdfs;3.10.4.7 from central in [default] 
     ai.h2o#h2o-persist-s3;3.10.4.7 from central in [default] 
     ai.h2o#h2o-scala_2.11;3.10.4.7 from central in [default] 
     ai.h2o#h2o-web;3.10.4.7 from central in [default] 
     ai.h2o#reflections;0.9.11-h2o-custom from central in [default] 
     ai.h2o#sparkling-water-core_2.11;2.1.7 from central in [default] 
     ai.h2o#sparkling-water-examples_2.11;2.1.7 from central in [default] 
     ai.h2o#sparkling-water-ml_2.11;2.1.7 from central in [default] 
     ai.h2o#sparkling-water-repl_2.11;2.1.7 from central in [default] 
     com.amazonaws#aws-java-sdk-core;1.10.47 from central in [default] 
     com.amazonaws#aws-java-sdk-kms;1.10.47 from central in [default] 
     com.amazonaws#aws-java-sdk-s3;1.10.47 from central in [default] 
     com.github.rwl#jtransforms;2.4.0 from central in [default] 
     com.google.code.findbugs#jsr305;3.0.0 from central in [default] 
     com.google.code.gson#gson;2.3.1 from central in [default] 
     com.google.guava#guava;19.0 from central in [default] 
     commons-codec#commons-codec;1.6 from central in [default] 
     commons-httpclient#commons-httpclient;3.1 from central in [default] 
     commons-io#commons-io;2.4 from central in [default] 
     commons-lang#commons-lang;2.6 from central in [default] 
     commons-logging#commons-logging;1.1.3 from central in [default] 
     gov.nist.math#jama;1.0.3 from central in [default] 
     joda-time#joda-time;2.8.1 from central in [default] 
     log4j#log4j;1.2.15 from central in [default] 
     net.java.dev.jets3t#jets3t;0.6.1 from central in [default] 
     net.java.dev.jna#jna;4.0.0 from central in [default] 
     net.sf.opencsv#opencsv;2.3 from central in [default] 
     org.apache.commons#commons-math3;3.3 from central in [default] 
     org.apache.httpcomponents#httpclient;4.3.6 from central in [default] 
     org.apache.httpcomponents#httpcore;4.3.3 from central in [default] 
     org.eclipse.jetty#jetty-continuation;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-http;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-io;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-jndi;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-plus;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-security;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-server;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-servlet;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-util;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-webapp;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty#jetty-xml;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty.aggregate#jetty-servlet;8.1.17.v20150415 from central in [default] 
     org.eclipse.jetty.orbit#javax.activation;1.1.0.v201105071233 from central in [default] 
     org.eclipse.jetty.orbit#javax.mail.glassfish;1.4.1.v201005082020 from central in [default] 
     org.eclipse.jetty.orbit#javax.servlet;3.0.0.v201112011016 from central in [default] 
     org.eclipse.jetty.orbit#javax.transaction;1.1.1.v201105210645 from central in [default] 
     org.javassist#javassist;3.18.2-GA from central in [default] 
     org.joda#joda-convert;1.7 from central in [default] 
     org.kohsuke#libpam4j;1.8 from central in [default] 
     :: evicted modules: 
     joda-time#joda-time;2.3 by [joda-time#joda-time;2.8.1] in [default] 
     org.apache.httpcomponents#httpclient;4.1 by [org.apache.httpcomponents#httpclient;4.3.6] in [default] 
     org.apache.httpcomponents#httpcore;4.1 by [org.apache.httpcomponents#httpcore;4.3.3] in [default] 
     commons-logging#commons-logging;1.1.1 by [commons-logging#commons-logging;1.1.3] in [default] 
     commons-codec#commons-codec;1.4 by [commons-codec#commons-codec;1.6] in [default] 
     com.google.guava#guava;16.0.1 by [com.google.guava#guava;19.0] in [default] 
     com.google.guava#guava;18.0 by [com.google.guava#guava;19.0] in [default] 
     commons-codec#commons-codec;1.3 by [commons-codec#commons-codec;1.4] in [default] 
     commons-logging#commons-logging;1.0.4 by [commons-logging#commons-logging;1.1.1] in [default] 
     commons-codec#commons-codec;1.2 by [commons-codec#commons-codec;1.4] in [default] 
     --------------------------------------------------------------------- 
     |     |   modules   || artifacts | 
     |  conf  | number| search|dwnlded|evicted|| number|dwnlded| 
     --------------------------------------------------------------------- 
     |  default  | 68 | 15 | 15 | 10 || 55 | 12 | 
     --------------------------------------------------------------------- 
:: retrieving :: org.apache.spark#spark-submit-parent 
     confs: [default] 
     12 artifacts copied, 43 already retrieved (23416kB/63ms) 
Traceback (most recent call last): 
    File "/home/hadoop/scripts/test/spark.py", line 3, in <module> 
    hc = H2OContext.getOrCreate(sc) 
NameError: name 'H2OContext' is not defined

來源

2017-06-03 Keston

的命令有2.1.7因爲這是我想的最後一件事，但其與2.1.8相同。 – Keston

還有另一個名爲pysparkling的軟件包與h2o不相關。當我點子安裝它，這是問題。我現在可以在沒有問題的集羣上運行它。終止該羣集並且僅使用h2o_pysparkling_2.1而不是pysparkling進行正確的引導。有效。同時安裝導致問題。 – Keston

你並不需要附加蘇打水包裝（選件--packages），但你需要提供pysparkling Python包（它包含內部所有必需的二進制依賴關係）。

最好的辦法是從http://h2o.ai/download直接下載蘇打水的二進制分發和使用bin/pysparkling腳本或使用火花：

$SPARK_HOME/bin/pyspark --py-files h2o_pysparkling_2.1-2.1.8.zip

來源

2017-06-05 03:11:50 Michal

謝謝！我測試了它，它工作。我正在尋找目錄中的.egg文件，這些文件在我猜想的更新版本中不再存在。 --package的作品也是，我的問題是我已經安裝了pysparkling在與h2o不相關的羣集上是一些隨機的其他包。使用pip安裝h2o_pysparkling_2.1的引導程序與--package arg一起工作。 – Keston

運行PySparkling與Spark 2.1的H20Context出錯

回答

相關問題