3
當我嘗試提交向kafka發送消息的spark任務時，我正在體驗OOME - 它將消息發送到Kafka（675字節） - 錯誤僅在執行程序即將關閉時顯示。Spark OutOfMemoryError
Diagnostics: Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
start time: 1441611385047
final status: FAILED
這裏的紗線日誌：
（1）：
INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
WARN thread.QueuedThreadPool: 7 threads could not be stopped
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-12"
Exception in thread "Thread-3"
（2）：
Exception in thread "shuffle-client-4" Exception in thread "shuffle-server-7"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "shuffle-client-4"
（3）：
INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
Exception in thread "LeaseRenewer:[email protected]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "LeaseRenewer:[email protected]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-akka.actor.default-dispatcher-16"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-akka.remote.default-remote-dispatcher-6"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "sparkDriver-akka.remote.default-remote-dispatcher-5"
Exception in thread "Thread-3"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-3"
在它顯示爲成功，但紗線日誌仍然有OOME罕見的情況：
INFO cluster.YarnClusterSchedulerBackend: Asking each executor to shut down
INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorActor: OutputCommitCoordinator stopped!
INFO spark.MapOutputTrackerMasterActor: MapOutputTrackerActor stopped!
INFO storage.MemoryStore: MemoryStore cleared
INFO storage.BlockManager: BlockManager stopped
INFO storage.BlockManagerMaster: BlockManagerMaster stopped
INFO spark.SparkContext: Successfully stopped SparkContext
INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
Exception in thread "Thread-3"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-3"
您可以添加相關代碼，以及其他相關信息，如羣集配置，驅動程序/工人內存，你在操作數據等 –
嗯的大小...的哪一部分內存被耗盡？ PermGen的？堆？ 也許試試暫時增加這兩個內存部分之一，看看問題出現在哪裏。如果它是PermGen - 也許你會加載太多的類定義？ – jarasss