elasticsearch索引性能不佳

目前，我使用彈性搜索來存儲和查詢一些日誌。我們建立了一個五節點彈性搜索集羣。其中兩個索引節點和三個查詢節點。在索引節點中，我們在兩臺服務器上都有redis，logstash和elasticsearch。 elasticsearch使用NFS存儲作爲數據存儲。我們的要求是索引300個日誌條目/秒。但是我可以從elasticsearch獲得的最佳性能僅爲25個日誌條目/秒！ elasticsearch的XMX是16G。版本每個組件的：elasticsearch索引性能不佳

Redis: 2.8.12 
logstash: 1.4.2 
elasticsearch: 1.5.0

我們目前的指數設置是這樣的：

 { 
     "userlog" : { 
     "settings" : { 
      "index" : { 
      "index" : { 
       "store" : { 
       "type" : "mmapfs" 
       }, 
       "translog" : { 
       "flush_threshold_ops" : "50000" 
       } 
      }, 
      "number_of_replicas" : "1", 
      "translog" : { 
       "flush_threshold_size" : "1G", 
       "durability" : "async" 
      }, 
      "merge" : { 
       "scheduler" : { 
       "max_thread_count" : "1" 
       } 
      }, 
      "indexing" : { 
       "slowlog" : { 
       "threshold" : { 
        "index" : { 
        "trace" : "2s", 
        "info" : "5s" 
        } 
       } 
       } 
      }, 
      "memory" : { 
       "index_buffer_size" : "3G" 
      }, 
      "refresh_interval" : "30s", 
      "version" : { 
       "created" : "1050099" 
      }, 
      "creation_date" : "1447730702943", 
      "search" : { 
       "slowlog" : { 
       "threshold" : { 
        "fetch" : { 
        "debug" : "500ms" 
        }, 
        "query" : { 
        "warn" : "10s", 
        "trace" : "1s" 
        } 
       } 
       } 
      }, 
      "indices" : { 
       "memory" : { 
       "index_buffer_size" : "30%" 
       } 
      }, 
      "uuid" : "E1ttme3fSxKVD5kRHEr_MA", 
      "index_currency" : "32", 
      "number_of_shards" : "5" 
      } 
     } 
     } 
    }

這裏是我的logstash配置：

input { 
      redis { 
        host => "eanprduserreporedis01.eao.abn-iad.ea.com" 
        port => "6379" 
        type => "redis-input" 
        data_type => "list" 
        key => "userLog" 
        threads => 15 
      } 
     # Second reids block begin 
      redis { 
        host => "eanprduserreporedis02.eao.abn-iad.ea.com" 
        port => "6379" 
        type => "redis-input" 
        data_type => "list" 
        key => "userLog" 
        threads => 15 
      } 
      # Second reids block end 
    } 

    output { 
      elasticsearch { 
        cluster => "customizedlog_prod" 
        index => "userlog" 
        workers => 30 
      } 
      stdout{} 
    }

一個非常奇怪的是，雖然目前的索引速度只有〜20/s，IO等待幾乎高達70％。主要是讀取流量。通過nfsiostat，目前的讀取速度約爲200Mbps！所以基本上，爲了索引每個日誌條目，它會讀取大約10Mbits的數據，這是瘋狂的，因爲我們的日誌條目的平均長度小於10K。所以，我採取了彈性搜索的jstack轉儲，這裏是一個正在運行的線程的結果：

"elasticsearch[somestupidhostname][bulk][T#3]" daemon prio=10 tid=0x00007f230c109800 nid=0x79f6 runnable [0x00007f1ba85f0000] 
     java.lang.Thread.State: RUNNABLE 
      at sun.nio.ch.FileDispatcherImpl.pread0(Native Method) 
      at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52) 
      at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220) 
      at sun.nio.ch.IOUtil.read(IOUtil.java:197) 
      at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:730) 
      at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:715) 
      at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179) 
      at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342) 
      at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:54) 
      at org.apache.lucene.store.DataInput.readVInt(DataInput.java:122) 
      at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:221) 
      at org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:152) 
      at org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:506) 
      at org.elasticsearch.common.lucene.uid.PerThreadIDAndVersionLookup.lookup(PerThreadIDAndVersionLookup.java:104) 
      at org.elasticsearch.common.lucene.uid.Versions.loadDocIdAndVersion(Versions.java:150) 
      at org.elasticsearch.common.lucene.uid.Versions.loadVersion(Versions.java:161) 
      at org.elasticsearch.index.engine.InternalEngine.loadCurrentVersionFromIndex(InternalEngine.java:1002) 
      at org.elasticsearch.index.engine.InternalEngine.innerCreate(InternalEngine.java:277) 
      - locked <0x00000005fc76b938> (a java.lang.Object) 
      at org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:256) 
      at org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:455) 
      at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:437) 
      at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:149) 
      at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:515) 
      at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:422) 
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
      at java.lang.Thread.run(Thread.java:745)

誰能告訴我什麼是彈性的搜索做，爲什麼索引是這麼慢？有可能改進它嗎？

來源

2015-12-24 張新峯

您有一些組件，每個組件都可能很慢。除非你設置了一個批量插入數據的獨立測試到elasticsearch，罪魁禍首可以很容易地logstash，redis等 –

另外，發佈您的logstash配置;我有一個理論。 –

@AlainCollins我在帖子中添加了配置。順便說一句，我不認爲Redis或logstash是問題。因爲通過iotop，我可以看到大多數IO被elasticsearch使用。 –

它可能不完全負責你的糟糕的表現，但檢查出redis的batch_size選項。我敢打賭，如果你一次從redis中提取超過1個文件，它會變得更好。

來源

2015-12-25 08:24:53

elasticsearch索引性能不佳

回答

相關問題