星火SBT編譯問題

在我的編輯，即使我按在src /主/資源文件夾Twitter的jar文件，但SBT編譯沒有接他們，並編譯和打包沒有錯誤，但在運行時給了我錯誤的「類未找到twitterUtils」星火SBT編譯問題

我的問題是，爲什麼SBT不包括在編譯從資源文件夾中的jar文件？

人們告訴我要做所有這些複雜的步驟來獲取Git實用程序，然後做一個sbt程序集，我做了，但因爲iam後面的代理Git不工作，即使所有的http_proxy設置。我也試過把這些twitter jar文件放在CLASSPATH中，沒有運氣。我堅持這個問題，所以任何幫助，高度讚賞。請參閱下面

[[email protected] TwitterPopularTags]# pwd 
/root/TwitterPopularTags 
[[email protected] TwitterPopularTags]# sbt compile 
[info] Set current project to TwitterPopularTags (in build file:/root/TwitterPopularTags/) 
[info] Updating {file:/root/TwitterPopularTags/}twitterpopulartags... 
[info] Resolving jline#jline;2.12.1 ... 
[info] Done updating. 
[info] Compiling 2 Scala sources to /root/TwitterPopularTags/target/scala-2.11/classes... 
[success] Total time: 14 s, completed Sep 16, 2016 9:55:20 AM 
[[email protected] TwitterPopularTags]# sbt package 
[info] Set current project to TwitterPopularTags (in build file:/root/TwitterPopularTags/) 
[info] Packaging /root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar ... 
[info] Done packaging. 
[success] Total time: 1 s, completed Sep 16, 2016 9:56:20 AM 
[[email protected] TwitterPopularTags]# spark-submit /root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar 
16/09/16 09:57:06 INFO SparkContext: Running Spark version 1.6.2 
16/09/16 09:57:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/09/16 09:57:06 INFO SecurityManager: Changing view acls to: root 
16/09/16 09:57:06 INFO SecurityManager: Changing modify acls to: root 
16/09/16 09:57:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 
16/09/16 09:57:07 INFO Utils: Successfully started service 'sparkDriver' on port 53967. 
16/09/16 09:57:07 INFO Slf4jLogger: Slf4jLogger started 
16/09/16 09:57:07 INFO Remoting: Starting remoting 
16/09/16 09:57:07 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:57877] 
16/09/16 09:57:07 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 57877. 
16/09/16 09:57:07 INFO SparkEnv: Registering MapOutputTracker 
16/09/16 09:57:07 INFO SparkEnv: Registering BlockManagerMaster 
16/09/16 09:57:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-47a89077-0926-447c-ada7-fdb4a9aa1b83 
16/09/16 09:57:07 INFO MemoryStore: MemoryStore started with capacity 511.5 MB 
16/09/16 09:57:07 INFO SparkEnv: Registering OutputCommitCoordinator 
16/09/16 09:57:08 INFO Server: jetty-8.y.z-SNAPSHOT 
16/09/16 09:57:08 INFO AbstractConnector: Started [email protected]:4040 
16/09/16 09:57:08 INFO Utils: Successfully started service 'SparkUI' on port 4040. 
16/09/16 09:57:08 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.100.44.17:4040 
16/09/16 09:57:08 INFO HttpFileServer: HTTP File server directory is /tmp/spark-d56628b6-fdbf-4d89-bbd2-a96603000607/httpd-ee499eb3-00ae-4276-b163-423e3b81f0b4 
16/09/16 09:57:08 INFO HttpServer: Starting HTTP Server 
16/09/16 09:57:08 INFO Server: jetty-8.y.z-SNAPSHOT 
16/09/16 09:57:08 INFO AbstractConnector: Started [email protected]:56067 
16/09/16 09:57:08 INFO Utils: Successfully started service 'HTTP file server' on port 56067. 
16/09/16 09:57:08 INFO SparkContext: Added JAR file:/root/TwitterPopularTags/target/scala-2.11/twitterpopulartags_2.11-1.0.jar at http://10.100.44.17:56067/jars/twitterpopulartags_2.11-1.0.jar with timestamp 1474034228091 
16/09/16 09:57:08 INFO Executor: Starting executor ID driver on host localhost 
16/09/16 09:57:08 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49715. 
16/09/16 09:57:08 INFO NettyBlockTransferService: Server created on 49715 
16/09/16 09:57:08 INFO BlockManagerMaster: Trying to register BlockManager 
16/09/16 09:57:08 INFO BlockManagerMasterEndpoint: Registering block manager localhost:49715 with 511.5 MB RAM, BlockManagerId(driver, localhost, 49715) 
16/09/16 09:57:08 INFO BlockManagerMaster: Registered BlockManager 
16/09/16 09:57:08 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 
16/09/16 09:57:08 INFO EventLoggingListener: Logging events to hdfs:///spark-history/local-1474034228122 
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/twitter/TwitterUtils$ 
     at dot.state.fl.us.PrintTweets$.main(PrintTweets.scala:29) 
     at dot.state.fl.us.PrintTweets.main(PrintTweets.scala)

來源

2016-09-16 Ahmad S

細節我的問題是，爲什麼SBT不包括在編譯從資源文件夾中的jar文件？

因爲這不是資源文件夾是。如果要手動管理依賴關係，請輸入put them into lib folder instead。但在這種情況下，您還需要對這些依賴關係的所有依賴關係，它們的依賴關係等進行相同操作。正如鏈接文檔中所述，使用託管依賴關係通常是一個更好的主意。

來源

2016-09-16 14:33:43

嗨Alexey我沒有看到鏈接的文檔可以請你張貼鏈接在這裏？爲什麼不SBT下載Twitter的依賴JAR文件，因爲它在build.sbt文件相關的列表「火花流，嘰嘰喳喳？請查看下面的」 libraryDependencies ++ = SEQ（ ‘org.apache.spark’％「spark-core」％sparkVersion，「」org.apache.spark「%%」spark-streaming「 –

對不起，我發現鏈接沒關係，但我的問題仍然存在。爲什麼Twitter的罐子不被下載，即使他們是在依賴關係列表？或者他們正在下載和我找不到類錯誤是什麼東西？ –

我將我的twitter jar文件放在lib文件夾和clean-compile-package中，但仍未找到「twitterUtils」[root @ hadoop1 lib]＃pwd /root/TwitterPopularTags/lib [root @ hadoop1 lib]＃ls -ltr total 360 -rw-r - r-- 1 root root 60808 Sep 16 11:18 twitter4j-stream-4.0.4.jar -rw-R - R-- 1根根290456 09月16日11時18分twitter4j核-4.0.4.ziptwitter4j核-4.0.4.jar -rw-R - R-- 1根根14115月16 11:18 dstream-twitter_2.11-0.1.0-SNAPSHOT.jar [root @ hadoop1 lib]＃ –

星火SBT編譯問題

回答

相關問題