引發內存不足錯誤

我的火花程序在小數據集上運行良好（約400GB）但是，當我將其擴展到大型數據集時。我開始得到錯誤java.lang.OutOfMemoryError: Java heap space或java.lang.OutOfMemoryError: Requested array size exceeds VM limit引發內存不足錯誤

我的計劃是這樣的： sc.textFile - >地圖 - >過濾器 - > GROUPBY - > saveAsObjectFile

groupBy生成類型RDD [（INT，可迭代[結果A]）]

發生錯誤saveAsObjectFile。我能想到的唯一原因是：在groupBy步驟一些鍵包含太大的數據。但是我用Hive檢查了所有密鑰，最大的一個是330808. class A也不是很大。

我的配置是： -driver內存20G --num執行人120 --executor內存30G 星火版本：1.4

15/07/03 07:05:06 ERROR ActorSystemImpl: Uncaught fatal error from thread 
[sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down ActorSystem [sparkDriver] 
java.lang.OutOfMemoryError: Java heap space 
     at java.util.Arrays.copyOf(Arrays.java:2271) 
     at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) 
     at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) 
     at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) 
     at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1876) 
     at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785) 
     at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188) 
     at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) 
     at akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply$mcV$sp(Serializer.scala:129) 
     at akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129) 
     at akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129) 
     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) 
     at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129) 
     at akka.remote.MessageSerializer$.serialize(MessageSerializer.scala:36) 
     at akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:845) 
     at akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:845) 
     at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) 
     at akka.remote.EndpointWriter.serializeMessage(Endpoint.scala:844) 
     at akka.remote.EndpointWriter.writeSend(Endpoint.scala:747) 
     at akka.remote.EndpointWriter$$anonfun$2.applyOrElse(Endpoint.scala:722) 
     at akka.actor.Actor$class.aroundReceive(Actor.scala:465) 
     at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415) 
     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) 
     at akka.actor.ActorCell.invoke(ActorCell.scala:487) 
     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) 
     at akka.dispatch.Mailbox.run(Mailbox.scala:220)

來源

2015-07-03 worldterminator

是來自驅動程序或執行程序的此錯誤消息嗎？ –

@VijayInnamuri問題已更新。似乎來自司機。 – worldterminator