我有一個簡單的節點腳本,它將把不存在的文檔從redis複製到elasticsearch。但是,我的腳本因內存失敗。我意識到Node.js有1G內存的限制。我對自己在Redis的文件1.7G(據Redis的)Node.js腳本從redis複製到elasticsearch內存不足
# Memory
used_memory:1828855608
used_memory_human:1.70G
used_memory_rss:1768804352
used_memory_peak:1839670312
used_memory_peak_human:1.71G
這裏是我的簡單的節點腳本
'use strict';
const redis = require('redis');
const elasticsearch = require('elasticsearch');
const config = require('../config');
let client = redis.createClient({
host: config.redis.url,
port: config.redis.port
});
let esClient = elasticsearch.Client({
host: config.elasticsearch.url
});
client.keys('*', (err, keys) => {
if (err) {
console.log(err);
}
keys.forEach((key) => {
esClient.exists({
index: 'articles',
type: 'article',
id: key
}, (err, exists) => {
if (!exists) {
}
});
});
});
這是我的錯誤。
<--- Last few GCs --->
56742 ms: Scavenge 1398.8 (1457.2) -> 1398.8 (1457.2) MB, 0.8/0 ms (+ 2.2 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
57796 ms: Mark-sweep 1398.8 (1457.2) -> 1398.8 (1457.2) MB, 1054.1/0 ms (+ 3.2 ms in 2 steps since start of marking, biggest step 2.2 ms) [last resort gc].
58835 ms: Mark-sweep 1398.8 (1457.2) -> 1398.8 (1457.2) MB, 1038.5/0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x220f52737399 <JS Object>
1: test [native regexp.js:~132] [pc=0x2b04c4f8bd00] (this=0xdea7c8c02e9 <JS RegExp>,k=0xdea7c8b0741 <String[24]: /articles/article/1HcdBc>)
2: new constructor(aka ClientRequest) [_http_client.js:~19] [pc=0x2b04c4fa5ebd] (this=0xdea7c8c0459 <a ClientRequest with map 0x11d7d52ec4f1>,options=0xdea7c8c0409 <an Object with map 0x11d7d52f6731>,cb=0xdea7c8c03c1 <JS Function (SharedFunctionInfo...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
sh: line 1: 96220 Abort trap: 6 node scripts/copy-to-es.js
然而,從Redis的是1.7G的文件的整個但我想我只是讓所有的鑰匙,檢查它是否存在於Elasticsearch與否。我不確定爲什麼它因記憶而失敗,我如何克服這一點?我正在使用Node 4.2.2
我正在使用10.1.3 – toy