管道中的Redis中的MemoryError

Question

我正在使用此python腳本將數據從一個ElastiCache redis實例遷移到另一個ElastiCache redis實例。它使用redis流水線以塊爲單位遷移數據。

https://gist.github.com/thomasst/afeda8fe80534a832607

但我得到這個奇怪的錯誤：

Traceback (most recent call last):###########           | ETA: 0:00:12 
    File "migrate-redis.py", line 95, in <module> 
    migrate() 
    File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 664, in __call__ 
    return self.main(*args, **kwargs) 
    File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 644, in main 
    rv = self.invoke(ctx) 
    File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 837, in invoke 
    return ctx.invoke(self.callback, **ctx.params) 
    File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 464, in invoke 
    return callback(*args, **kwargs) 
    File "migrate-redis.py", line 74, in migrate 
    results = pipeline.execute(False) 
    File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2593, in execute 
    return execute(conn, stack, raise_on_error) 
    File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 2446, in _execute_transaction 
    all_cmds = connection.pack_commands([args for args, _ in cmds]) 
    File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 637, in pack_commands 
    output.append(SYM_EMPTY.join(pieces)) 
MemoryError 

沒有與RAM沒有問題的節點有6 GB的RAM。

源的Redis的

記憶概況如下：

used_memory:1483900120 
used_memory_human:1.38G 
used_memory_rss:1945829376 
used_memory_peak:2431795528 
used_memory_peak_human:2.26G 
used_memory_lua:86016 
mem_fragmentation_ratio:1.31 
mem_allocator:jemalloc-3.6.0 

什麼可以是可能的原因呢？

Answer 1

從您的錯誤日誌中，它與您的redis服務器沒有任何關係。將所有命令打包到內存緩衝區時，錯誤發生在您的redis客戶端中。

也許你可以嘗試減少migrate-redis.py中的掃描計數選項，以測試它是否太大而無法打包它。