1. 首頁
  2. 問答
  3. 在spark-shell中導入包時出錯

在spark-shell中導入包時出錯

2017-10-15 109 views
1

對不起，這個問題似乎不時問到。我已經閱讀了我所見過的答案，但由於我是scala新手，我可能還沒有理解他們。在spark-shell中導入包時出錯

我的.jar我想在火花外殼使用構建，所以我推出殼基科這樣的：

$ spark-shell --driver-memory 1g --jars lsa.jar

一切似乎是確定：

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 
17/10/15 16:57:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
17/10/15 16:57:59 WARN Utils: Your hostname, jice-H81M-HD3 resolves to a loopback address: 127.0.1.1; using 192.168.0.11 instead (on interface enp3s0) 
17/10/15 16:57:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 
17/10/15 16:58:04 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException 
Spark context Web UI available at http://192.168.0.11:4040 
Spark context available as 'sc' (master = local[*], app id = local-1508079480365). 
Spark session available as 'spark'. 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.2.0 
     /_/ 

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144) 
Type in expressions to have them evaluated. 
Type :help for more information.

但是，當我嘗試使用它，IHAVE錯誤：

scala> import com.cloudera.datascience.lsa._ 
<console>:23: error: object cloudera is not a member of package com 
     import com.cloudera 
      ^

我在這裏看到：Apache Spark not recognising import from external Jar到org.apache.spa參考rk.repl.SparkILoop＃require（）但我無法弄清楚如何使用它。

有人能給我一個線索嗎？

謝謝

PS。 jar -tvf lsa.jar |的結果grep的LSA

 0 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ 
    2485 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$merge$1$1.class 
    2476 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$8$$anonfun$10.class 
    3926 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsInTopConcepts$1.class 
    1307 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$trimLeastFrequent$1.class 
    1391 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$7.class 
    2250 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$saveDocFreqs$2.class 
    1283 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$13.class 
    2071 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsInTopConcepts$1$$anonfun$apply$6.class 
    1344 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$row$3.class 
    1442 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$2$$anonfun$3.class 
    1842 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$documentFrequencies$2.class 
    1373 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$2$$anonfun$apply$3.class 
    2816 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$2.class 
    1261 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$10.class 
    1752 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$2.class 
    2702 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$multiplyByDiagonalMatrix$2.class 
    1721 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$multiplyByDiagonalMatrix$2$$anonfun$1.class 
    1557 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$5.class 
    1351 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$5.class 
    1474 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$1.class 
    1273 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsForTerm$1.class 
    2317 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$6$$anonfun$apply$5.class 
    1841 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$documentFrequencies$1.class 
15390 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$.class 
    2897 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$1.class 
    3454 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$8.class 
    1300 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$11.class 
    1668 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$1$$anonfun$2.class 
    1312 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$saveDocFreqs$1.class 
    1877 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$inverseDocumentFrequencies$1.class 
    1371 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$12.class 
    1152 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$4.class 
    1497 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$comb$1$1.class 
    3952 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsInTopConcepts$1.class 
    2241 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$plainTextToLemmas$1.class 
    1332 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$14.class 
    1373 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$2$$anonfun$apply$4.class 
    1365 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$12.class 
    2500 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$plainTextToLemmas$1$$anonfun$apply$4.class 
    1416 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$2$$anonfun$apply$2.class 
10049 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA.class 
    1401 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$11.class 
    1491 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsForDoc$1.class 
    1726 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$printIdWeights$1.class 
    1331 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$2$$anonfun$3$$anonfun$apply$1.class 
    2141 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$8$$anonfun$9.class 
    2055 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsInTopConcepts$1$$anonfun$apply$7.class 
    2678 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$main$2.class 
    2093 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$6.class 
    1742 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$rowsNormalized$1$$anonfun$apply$1.class 
    1214 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$1.class 
    1430 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$comb$1$2$$anonfun$apply$3.class 
    1389 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$termDocumentMatrix$1.class 
    1363 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$13.class 
    1839 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$multiplyByDiagonalMatrix$1.class 
    2678 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$2$$anonfun$3.class 
    1439 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$merge$1$1$$anonfun$apply$2.class 
    1414 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$row$2.class 
    1399 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$readFile$1.class 
    1320 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$row$1.class 
    1562 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$6.class 
20269 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$.class 
    2759 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$com$cloudera$datascience$lsa$ParseWikipedia$$comb$1$2.class 
    1651 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topDocsInTopConcepts$1$$anonfun$9.class 
    1616 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/RunLSA$$anonfun$topTermsInTopConcepts$1$$anonfun$8.class 
    7015 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia.class 
    1562 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$7.class 
    1422 Sun Oct 15 16:45:50 CEST 2017 com/cloudera/datascience/lsa/ParseWikipedia$$anonfun$4.class 
    0 Sun Oct 15 16:45:50 CEST 2017 META-INF/maven/com.cloudera.datascience/lsa/ 
    2152 Mon Mar 16 12:55:28 CET 2015 META-INF/maven/com.cloudera.datascience/lsa/pom.xml 
    113 Sun Oct 15 16:45:50 CEST 2017 META-INF/maven/com.cloudera.datascience/lsa/pom.properties

來源

2017-10-15 Jice

+0

你嘗試用完整路徑罐子？ –

+0

你可以做'jar -tvf lsa.jar | grep lsa'並粘貼輸出到你的問題？如果jar沒有找到，你會得到'FileNotFoundException'，我假設你沒有看到異常，不是嗎？ –

+0

謝謝。我今天晚上會這樣做。這個罐子很可能已經找到了，因爲我測試了帶有spark-shell --jars選項的lauch，指出了一條不好的路徑，並且當路徑正確時我沒有這個例外。 – Jice

回答

1

隨着--driver級的路徑，而不是--jars，它的工作原理

來源

2017-10-23 19:53:54 Jice

相關問題

 相關問題