爲什麼集羣模式比Spark中的客戶端模式慢得多？

Question

我有使用GraphFrame創建的15k圖形對象。這些圖很小（10-100個節點，每個圖10-100個邊）。我只需要在每個圖上應用最短路徑和某些其他算法。我目前的實現是使用for循環遍歷所有15k圖形。令人驚訝的是，集羣模式比客戶端模式慢得多。

我處理了10張圖。以下是客戶端模式和集羣模式的性能指標。我試圖找出爲什麼集羣模式比客戶端模式慢。我無法理解指標。

配置爲羣集（火花獨立）

配置爲客戶端模式：

客戶端模式：

Scheduling mode = FIFO 
Spark Context default degree of parallelism = 4 
Aggregated Spark stage metrics: 
numStages => 1019 
sum(numTasks) => 3263 
elapsedTime => 38675 (39 s) 
sum(stageDuration) => 19177 (19 s) 
sum(executorRunTime) => 41413 (41 s) 
sum(executorCpuTime) => 6305 (6 s) 
sum(executorDeserializeTime) => 6386 (6 s) 
sum(executorDeserializeCpuTime) => 3382 (3 s) 
sum(resultSerializationTime) => 82 (82 ms) 
sum(jvmGCTime) => 1171 (1 s) 
sum(shuffleFetchWaitTime) => 3 (3 ms) 
sum(shuffleWriteTime) => 603 (0.6 s) 
max(resultSize) => 23947 (23.0 KB) 
sum(numUpdatedBlockStatuses) => 1848 
sum(diskBytesSpilled) => 0 (0 Bytes) 
sum(memoryBytesSpilled) => 0 (0 Bytes) 
max(peakExecutionMemory) => 35651584 
sum(recordsRead) => 16972 
sum(bytesRead) => 22064469 (21.0 MB) 
sum(recordsWritten) => 0 
sum(bytesWritten) => 0 (0 Bytes) 
sum(shuffleTotalBytesRead) => 1431309 (1397.0 KB) 
sum(shuffleTotalBlocksFetched) => 4304 
sum(shuffleLocalBlocksFetched) => 4304 
sum(shuffleRemoteBlocksFetched) => 0 
sum(shuffleBytesWritten) => 1205067 (1176.0 KB) 
sum(shuffleRecordsWritten) => 8228 

集羣模式

Scheduling mode = FIFO 
Spark Context default degree of parallelism = 16 
Aggregated Spark stage metrics: 
numStages => 1019 
sum(numTasks) => 3371 
elapsedTime => 119895 (2.0 min) 
sum(stageDuration) => 114501 (1.9 min) 
sum(executorRunTime) => 162486 (2.7 min) 
sum(executorCpuTime) => 29376 (29 s) 
sum(executorDeserializeTime) => 74342 (1.2 min) 
sum(executorDeserializeCpuTime) => 11919 (12 s) 
sum(resultSerializationTime) => 87 (87 ms) 
sum(jvmGCTime) => 7422 (7 s) 
sum(shuffleFetchWaitTime) => 20899 (21 s) 
sum(shuffleWriteTime) => 1144 (1 s) 
max(resultSize) => 61683 (60.0 KB) 
sum(numUpdatedBlockStatuses) => 6097 
sum(diskBytesSpilled) => 0 (0 Bytes) 
sum(memoryBytesSpilled) => 0 (0 Bytes) 
max(peakExecutionMemory) => 47185920 
sum(recordsRead) => 16962 
sum(bytesRead) => 22037962 (21.0 MB) 
sum(recordsWritten) => 0 
sum(bytesWritten) => 0 (0 Bytes) 
sum(shuffleTotalBytesRead) => 1480503 (1445.0 KB) 
sum(shuffleTotalBlocksFetched) => 4571 
sum(shuffleLocalBlocksFetched) => 2361 
sum(shuffleRemoteBlocksFetched) => 2210 
sum(shuffleBytesWritten) => 1211392 (1183.0 KB) 
sum(shuffleRecordsWritten) => 8228 

Answer 1

TL; DR您正在使用的工作錯誤的工具。

非常小的圖（10-100個節點，每個圖的10-100個邊）完全不受益於分佈式處理。

同時，像Spark這樣的分佈式系統中使用的高成本抽象增加了大量的開銷，使得它佔據了成本。

一般來說：

• 分佈式圖處理是困難的。
• 在大多數情況下，擴大規模都會擴大規模。