Econn Node.js客戶端與elasticsearch連接時，Econn Reset和Heap內存空間不足

我有一個解析70個日誌文件（每個文件大約15 MB）的代碼。我從日誌中提取特定的字段並向彈性搜索發送批量請求。以下是代碼。Econn Node.js客戶端與elasticsearch連接時，Econn Reset和Heap內存空間不足

//looks for all the files with .log extension 

glob(__dirname + "/../logs/*.log", function (er, files) { 
for (var i = 0; i < files.length; i++) { 

    lr = new LineByLineReader(files[i]); 
    lr.on('error', function (err) { 
     console.error(err.stack); 
    }); 

//each line of the file is taken and certain fields are taken from it and pushed into an array 
    lr.on('line', function (line) { 
     arr = line.replace(/['"]+/g, '').split(" "); 
     var jsonArg1 = new Object(); 
     var temparr = arr[13].split("?"); 
     jsonArg1.url = temparr[0]; 
     jsonArg1.method = arr[12]; 
     pluginArrayArg.push(jsonArg1); 
    }); 
//once the entire file is parsed, the array is pushed into elasticsearch bulk api 

    lr.on('end', function() { 
     // All lines are read, file is closed now. 
     data = (function() { 
      var x = ''; 
      for (var i = 0; i < pluginArrayArg.length; i++) { 
       x = x + '{ "index" : { "_index" : "nodeclient" , "_type": "logs"} }\n' + '{"method" : "' + pluginArrayArg[i].method + '", "url" : "' + pluginArrayArg[i].url + '"}\n'; 
      } 
      return x; 
     })(); 

     client.bulk({ 
      body: [ 
        data 
        ] 
     }, function (err, resp) { 
      if (err) { 
       console.log(err); 
      } 
      console.log(resp); 
      //console.log(JSON.stringify(resp)); 
     }); 
    }); 
    } 
});

的代碼執行精細周邊的7〜8的日誌文件，之後，我開始得到下面的錯誤反覆

Elasticsearch ERROR: 2016-10-28T21:09:21Z 
    Error: Request error, retrying 
    POST http:/localhost:9200/_bulk => read ECONNRESET 
     at Log.error (/home/apitestcoverage/apicoverage_node/node_modules/elasticsearch/src/lib/log.js:225:56) 
     at checkRespForFailure (/home/apitestcoverage/apicoverage_node/node_modules/elasticsearch/src/lib/transport.js:240:18) 
     at HttpConnector.<anonymous> (/home/apitestcoverage/apicoverage_node/node_modules/elasticsearch/src/lib/connectors/http.js:162:7) 
     at ClientRequest.wrapper (/home/apitestcoverage/apicoverage_node/node_modules/lodash/index.js:3095:19) 
     at emitOne (events.js:96:13) 
     at ClientRequest.emit (events.js:188:7) 
     at Socket.socketErrorListener (_http_client.js:308:9) 
     at emitOne (events.js:96:13) 
     at Socket.emit (events.js:188:7) 
     at emitErrorNT (net.js:1271:8)

然後最後下面致命erorr堆出空間錯誤信息的出現，並然後退出

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

我曾嘗試增加類似下面的命令行中的堆空間，仍然無法正常工作

節點--max_old_space_size = 4096 server.js

單獨搜索這些錯誤，人們談論增加maxSockets和重新使用池連接。我不知道對我的代碼進行這些更改。

我該如何着手解決這些錯誤？這兩個錯誤都是相關的嗎？

來源

2016-10-28 Sivakumar Sivaraman

我剛剛得到econnreset錯誤使用我的自定義批量插入腳本批量許多25-40mb文件。

我將虛擬盒安裝上的內存從4mb增加到8mb，我不再有這個問題。

來源

2016-12-07 13:54:04

Econn Node.js客戶端與elasticsearch連接時，Econn Reset和Heap內存空間不足

回答

相關問題