的Java星火蔓延內存映射的OutOfMemoryError錯誤

Question

我運行下面的代碼（包含約10000行）：

IndexedRowMatrix qTmp = new IndexedRowMatrix(a.rows()); 
IndexedRowMatrix qTmpT = qTmp.toCoordinateMatrix().transpose().toIndexedRowMatrix(); 

我得到以下錯誤：

15/06/19 10:08:52 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (1 time so far) 
15/06/19 10:08:54 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (2 times so far) 
15/06/19 10:08:57 INFO ExternalAppendOnlyMap: Thread 66 spilling in-memory map of 24.8 MB to disk (3 times so far) 
15/06/19 10:09:02 ERROR Executor: Exception in task 0.0 in stage 9.0 (TID 9) 
java.lang.OutOfMemoryError: Java heap space 

我曾嘗試調諧內存與各種參數以及：

SparkConf conf = new SparkConf(); 
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); 
conf.setAppName("test"); 
conf.setMaster("local[2]"); 
conf.set("spark.executor.memory", "6g"); 
conf.set("spark.driver.maxResultSize", "1g"); 
conf.set("spark.shuffle.spill", "false"); 
conf.set("spark.storage.memoryFraction", "0.3"); 
conf.set("spark.eventLog.enabled", "false"); 
conf.set("driver-memory", "4g")`; 
JavaSparkContext sc = new JavaSparkContext(conf); 

是否有一個原因，我一直得到這個溢出錯誤？

Answer 1

您應該使用重寫的java選項運行您的應用程序。

開始與-Xms程序= [大小] -Xmx -XX：MaxPermSize參數= [大小] -XX：MaxNewSize = [大小]

例如 -

-Xms512m -Xmx1152m -XX： MaxPermSize = 256m -XX：MaxNewSize = 256m