DSE Solr：併發查詢請求期間的Netty錯誤

Question

我有一個小型（5節點）的集羣，我正在使用它來基於各種特定索引策略對DSE搜索性能進行基準測試（我在CentOS上使用DSE 4.5.2） 。當我一次只提交一個查詢時，一切工作正常，但當我增加負載時，我很快會看到netty異常。在SolrJ（我大部分的測試已經使用Java客戶端已經），這些表現爲

Error with server [MY URL]: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: address already in use by: [id: 0x2d3873c4, local:E:2d3873c4 => local:local] 

這似乎發生持續每當我有多個線程併發查詢集羣，無論哪個節點實際上，我指着的並且不管請求是否來自同一臺服務器。在使用其他客戶端時（例如PySolr4），我也能夠引發這種行爲，所以這可能是DSE設置問題，而不是我的測試應用程序中的一些奇怪問題。

此外，如果我將dse.yaml中的節點間通信類型從netty更改爲http，則同一羣集將愉快地響應提交數百個併發請求的客戶端。我最初在dse.yaml中保留了默認的netty *設置，但遇到錯誤後，我玩了幾個選項（max_connections，acceptor/worker threads等）;不幸的是，我還沒有發現關於這些值的大量文檔，我所做的任何改變似乎都沒有達到預期效果。

從概念上講，似乎使用netty會提供比http更好的性能，所以我想試着弄清楚這裏可能會出現什麼問題。提前感謝您的任何建議。

見下面棧跟蹤從/var/log/cassandra/system.log對應於這些錯誤之一：

ERROR [http-8983-exec-4] 2014-11-19 18:35:51,557 SolrDispatchFilter.java (line 696) Error request exception: address already in use by: [id: 0x17617168, local:E:17617168 => local:local] 
java.lang.RuntimeException: address already in use by: [id: 0x17617168, local:E:17617168 => local:local] 
    at com.datastax.bdp.search.solr.handler.shard.netty.ShardClient.tryNewConnection(ShardClient.java:204) 
    at com.datastax.bdp.search.solr.handler.shard.netty.ShardClient.sendTo(ShardClient.java:163) 
    at com.datastax.bdp.search.solr.handler.shard.netty.NettyShardHandler.submit(NettyShardHandler.java:76) 
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:287) 
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:137) 
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1889) 
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:723) 
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:185) 
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413) 
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) 
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:147) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:218) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:100) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) 
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) 
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) 
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) 
    at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:891) 
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:750) 
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2283) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
    at java.lang.Thread.run(Unknown Source) 
Caused by: io.netty.channel.ChannelException: address already in use by: [id: 0x17617168, local:E:17617168 => local:local] 
    at io.netty.channel.local.LocalChannelRegistry.register(LocalChannelRegistry.java:46) 
    at io.netty.channel.local.LocalChannel.doBind(LocalChannel.java:178) 
    at io.netty.channel.local.LocalChannel$LocalUnsafe.connect(LocalChannel.java:349) 
    at io.netty.channel.DefaultChannelPipeline$HeadHandler.connect(DefaultChannelPipeline.java:1008) 
    at io.netty.channel.DefaultChannelHandlerContext.invokeConnect(DefaultChannelHandlerContext.java:495) 
    at io.netty.channel.DefaultChannelHandlerContext.connect(DefaultChannelHandlerContext.java:480) 
    at io.netty.channel.DefaultChannelHandlerContext.connect(DefaultChannelHandlerContext.java:465) 
    at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:847) 
    at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:199) 
    at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:165) 
    at io.netty.channel.local.LocalEventLoop.run(LocalEventLoop.java:33) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) 
    ... 1 more 

Answer 1

似乎是一個的Netty/JVM錯誤，涉及：https://github.com/netty/netty/issues/1765。你可以嘗試其他操作系統和/或JVM版本嗎？