「打開的文件太多」而傾倒數據elasticsearch

Question

我進行轉儲OSX上使用小牛以Elasticsearch-Exporter elasticsearch：

node /usr/bin/node_modules/elasticsearch-exporter/exporter.js -j ${esIndexName} -f esbackup

我有一個運行兩個節點，一個應用程序，它隨着應用程序節點總共添加了三個節點。由elasticsearch命令創建的節點是主節點。當我運行對我的索引export命令，我得到這個成功加載的幾秒鐘後：

2014-05-07T14:31:38.325-0700 [elasticsearch[Rancor][[es][1]: Lucene Merge Thread #0]] [WARN] merge.scheduler [][] - [Rancor] [es][1] failed to merge 
815 java.io.FileNotFoundException: /private/var/data/core/elasticsearch_me/nodes/0/indices/es/1/index/_f_es090_0.tip (Too many open files) 

我已經試過如下：

launchctl limit 10000

sudo launchctl limit 40000 65000

elasticsearch soft nofile 32000

elasticsearch hard nofile 32000

將-XX:-MaxFDLimit添加到我的應用程序的jvm參數

這些都不能解決我的問題。有時候加載完成時沒有錯誤，但大部分時間我都會遇到錯誤。有沒有人對我的問題可能有什麼想法/提示？

編輯：

$ launchctl limit cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 709 1064
maxfiles 10000 10240

$ sudo launchctl limit Password: cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 709 1064
maxfiles 40000 65000

Answer 1

好 - 如果你正在運行多個elasticsearch節點外加一個Mac上的Node.js應用程序，我肯定會做出一定的，你打開的文件數碰到ES推薦的限制：

file descriptor 

Make sure to increase the number of open files descriptors on the machine (or for the user running elasticsearch). Setting it to 32k or even 64k is recommended. 

In order to test how many open files the process can open, start it with -Des.max-open-files set to true. This will print the number of open files the process can open on startup. 

Alternatively, you can retrieve the max_file_descriptors for each node using the Nodes Info API, with: 

curl localhost:9200/_nodes/process?pretty 

您需要確定這是爲t他用戶運行ES，而不僅僅是root（除非你以root身份運行它）。

要做到這一點，我會遵循以下指示：（http://elasticsearch-users.115913.n3.nabble.com/quot-Too-many-open-files-quot-error-on-Mac-OSX-td4034733.html）。假設您需要32k和64k作爲限制：

In /etc/launchd.conf put: 

limit maxfiles 32000 64000 


Make sure in your ~/.bashrc file you are not setting the ulimit with something like "ulimit -n 1024". 

Open a new terminal, and run: 

launchctl limit maxfiles 
ulimit -a 

不要忘記在進行這些更改後重新啓動。然後，當你在這個命令行啓動參數elasticsearch通：

elasticsearch -XX:-MaxFDLimit 

後我的Mac上面的步驟我碰到Elasticsearch以下回應：

curl http://localhost:9200/_nodes/process?pretty 

{ 
    "cluster_name" : "elasticsearch", 
    "nodes" : { 
    "XXXXXXXXXXXXXXXXXXXXXXX" : { 
     "name" : "Marrina Smallwood", 
     "transport_address" : "inet[XX.XX.XX.XX:9300]", 
     "host" : "MacBook-Pro-Retina.local", 
     "ip" : "XX.XX.XX.XX", 
     "version" : "1.1.1", 
     "build" : "f1585f0", 
     "http_address" : "inet[/XX.XX.XX.XX:9200]", 
     "process" : { 
     "refresh_interval" : 1000, 
     "id" : 538, 
     "max_file_descriptors" : 32768, 
     "mlockall" : false 
     } 
    } 
    } 
}