分區後彈簧批量掛起

我正在處理200萬條記錄和20個網格大小，以供我的彈簧批量應用程序使用。對於100-200k的記錄，它工作正常，但是在200萬次分區後失敗。分區後彈簧批量掛起

分區信息：

{ 
    partition - 20 = { 
     startKey = 1980865.0, 
     endKey = 2069192.0, 
     partitionSize = 72, 
     partitionId = 20, 
     partitionRecordCount = 88328 
    }, 
    partition - 10 = { 
     st 
     artKey = 938305.0, 
     endKey = 1042560.0, 
     partitionSize = 72, 
     partitionId = 10, 
     partitionRecordCount = 104256 
    }, 
    partition - 11 = { 
     startKey = 1042561.0, 
     endKey = 1146816.0, 
     partitionSize = 72, 
     partitionId = 11, 
     partitionRecordCount = 104256 
    }, 
    partition - 12 = { 
     startKey = 1146817.0, 
     endKey = 1251072.0, 
     partitionSize = 72, 
     partitionId = 12, 
     partitionRecordCount = 104256 
    }, 
    partition - 13 = { 
     startKey = 1251073.0, 
     endKey = 1355328.0, 
     partitionSize = 72, 
     partitionId = 13, 
     partitionRecordCount = 104256 
    }, 
    par 
    tition - 14 = { 
     startKey = 1355329.0, 
     endKey = 1459584.0, 
     partitionSize = 72, 
     partitionId = 14, 
     partitionRecordCount = 104256 
    }, 
    partition - 15 = { 
     startKey = 1459585.0, 
     end 
     Key = 1563840.0, 
     partitionSize = 72, 
     partitionId = 15, 
     partitionRecordCount = 104256 
    }, 
    partition - 16 = { 
     startKey = 1563841.0, 
     endKey = 1668096.0, 
     partitionSize = 72, 
     p 
     artitionId = 16, 
     partitionRecordCount = 104256 
    }, 
    partition - 17 = { 
     startKey = 1668097.0, 
     endKey = 1772352.0, 
     partitionSize = 72, 
     partitionId = 17, 
     partitionRecordCoun 
     t = 104256 
    }, 
    partition - 18 = { 
     startKey = 1772353.0, 
     endKey = 1876608.0, 
     partitionSize = 72, 
     partitionId = 18, 
     partitionRecordCount = 104256 
    }, 
    partition - 19 = { 
     startKey = 
      1876609.0, 
     endKey = 1980864.0, 
     partitionSize = 72, 
     partitionId = 19, 
     partitionRecordCount = 104256 
    }, 
    partition - 1 = { 
     startKey = 1.0, 
     endKey = 104256.0, 
     partitionSize 
      = 72, 
     partitionId = 1, 
     partitionRecordCount = 104256 
    }, 
    partition - 2 = { 
     startKey = 104257.0, 
     endKey = 208512.0, 
     partitionSize = 72, 
     partitionId = 2, 
     partitionRecordCou 
     nt = 104256 
    }, 
    partition - 3 = { 
     startKey = 208513.0, 
     endKey = 312768.0, 
     partitionSize = 72, 
     partitionId = 3, 
     partitionRecordCount = 104256 
    }, 
    partition - 4 = { 
     startKey = 3127 
      69.0, 
     endKey = 417024.0, 
     partitionSize = 72, 
     partitionId = 4, 
     partitionRecordCount = 104256 
    }, 
    partition - 5 = { 
     startKey = 417025.0, 
     endKey = 521280.0, 
     partitionSize = 7 
      2, 
     partitionId = 5, 
     partitionRecordCount = 104256 
    }, 
    partition - 6 = { 
     startKey = 521281.0, 
     endKey = 625536.0, 
     partitionSize = 72, 
     partitionId = 6, 
     partitionRecordCount 
      = 104256 
    }, 
    partition - 7 = { 
     startKey = 625537.0, 
     endKey = 729792.0, 
     partitionSize = 72, 
     partitionId = 7, 
     partitionRecordCount = 104256 
    }, 
    partition - 8 = { 
     startKey = 729793 
      .0, 
     endKey = 834048.0, 
     partitionSize = 72, 
     partitionId = 8, 
     partitionRecordCount = 104256 
    }, 
    partition - 9 = { 
     startKey = 834049.0, 
     endKey = 938304.0, 
     partitionSize = 72, 
     partitionId = 9, 
     partitionRecordCount = 104256 
    } 
} 
Total no. of Partitions: 20

最後登錄它打印是創建的分區，然後控制從來沒有進入作家。對於200k條記錄來說，它很掙扎，所以我增加了連接池的大小，這解決了問題，但是它失敗了200萬行。

沒有錯誤，只是掛在那裏。

來源

2017-08-26 ViS

對於20個分區？你有沒有分析應用程序？它掛在什麼地方？我知道在使用大量分區導致應用程序看起來像掛起時，步驟執行存儲在作業存儲庫中的方式往往存在問題，因爲它們需要在工作人員啓動之前存儲。雖然... –

@MichaelMinella，我還沒有完成分析，但是分區是早些時候20分鐘，後來，我減少到5分鐘。它仍然等待4-5小時，以便作者開始寫作很少的文件，然後掛起或有時會因'通信鏈接失敗'錯誤而中斷。這個錯誤雖然從來沒有伴隨更少的數據。 – ViS

得到它的工作，分區查詢是不正確的，再加上，文件寫入沒有線程。 – ViS

這可能與db有關。你可以檢查應用程序是否能夠使用寫入憑據訪問數據庫。我注意到這種問題，分區已經完成，並且實際的處理被掛起。故障排除後，意識到它無法寫入數據庫。一旦db被修復，它運行良好。

來源

2017-11-20 18:31:37 Sowjanya

分區後彈簧批量掛起

回答

相關問題