2017-06-06 85 views
0

由於OutOfMemory錯誤,我們的solr會不時崩潰。我們在4.0.0版本上仍然存在問題,但在解決以下問題後計劃遷移到最新版本。內存不足(自動完成)內存不足

當我看着Tomcat的日誌我看到下列錯誤:

SEVERE: null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space 
    at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:469) 
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:297) 
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) 
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) 
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) 
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) 
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) 
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) 
    at java.lang.Thread.run(Thread.java:744) 
Caused by: java.lang.OutOfMemoryError: Java heap space 
    at org.apache.lucene.search.FieldComparator$TermOrdValComparator.<init>(FieldComparator.java:1124) 
    at org.apache.lucene.search.SortField.getComparator(SortField.java:425) 
    at org.apache.lucene.search.FieldValueHitQueue$MultiComparatorsFieldValueHitQueue.<init>(FieldValueHitQueue.java:110) 
    at org.apache.lucene.search.FieldValueHitQueue.create(FieldValueHitQueue.java:173) 
    at org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:1123) 
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:552) 
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:507) 
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:484) 
    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) 
    at si.amebis.termania.solr.ExternalSearch.search(ExternalSearch.java:307) 
    at si.amebis.termania.solr.ExternalSearch.handleRequestBody(ExternalSearch.java:235) 
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) 
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699) 
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455) 
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276) 
    ... 12 more 

剛過了自動完成字段的請求(建議您鍵入)。請求詳情如下:

q - *:* 
start - 0 
rows - 0 
fq - (Type:1 OR Type:2) 
facet - true 
facet.limit - 20 
facet.mincount - 1 
facet.sort - true 
facet.prefix - "mi" 
facet.field - "Autocomplete" 
-- 
which returns 8105170 hits 

其中自動填充字段被定義爲:

<field name="Autocomplete" type="grams" indexed="true" stored="false" omitNorms="true" required="False" multiValued="true" /> 
    <fieldtype name="grams" class="solr.TextField" positionIncrementGap="100"> 
     <analyzer type="index"> 
     <tokenizer class="solr.StandardTokenizerFactory" /> 
     <filter class="solr.ShingleFilterFactory" maxShingleSize="10" outputUnigrams="true" /> 
     <filter class="solr.LowerCaseFilterFactory" /> 
     <filter class="solr.TrimFilterFactory" /> 
     </analyzer> 
     <analyzer type="query"> 
     <tokenizer class="solr.StandardTokenizerFactory" /> 
     <filter class="solr.LowerCaseFilterFactory" /> 
     <filter class="solr.TrimFilterFactory" /> 
     </analyzer> 
    </fieldtype> 

指數細節:

Num document: 4338603 
Index size: 10.1 Gb 
Ram: 64Gb (-Xmx45000M) 
Terms count in Autocomplete field: 70.459.723 

我假設上的文本字段等許多方面刻面需要大量的的記憶。

如何計算需要多少內存?是否有更有效的方式提供自動完成(使用短語 - 正克)?

在此先感謝!

回答

0

你能連接連接到Solr實例來檢查內存在哪裏嗎?我猜這是在FieldCache,但總是很好的檢查,以確保,Solr的faceting分別對待每個領域,所以你應該能夠檢查特定領域的內存消耗。爲了估計內存使用量的方面查詢,你可以檢查此線程(http://lucene.472066.n3.nabble.com/Solr-using-a-ridiculous-amount-of-memory-td4050840.html

也有一些是還你的問題,你說你的查詢返回8105170命中,但你的指數只有4338603文件。通常在文本字段上刻面是具有挑戰性的,因爲術語的數量可能會增加得非常快,特別是如果使用帶狀皰疹/ ngrams的話。

看看https://github.com/cominvent/autocomplete是Solr支持的自動完成功能的一個很好的起點(我已經用它作爲我的幾個項目的出發點)。

根據您如何實現自動完成功能,您還可以嘗試更改facet.methodhttps://cwiki.apache.org/confluence/display/solr/Faceting )參數並檢查它是否有幫助。

也看看https://cwiki.apache.org/confluence/display/solr/Suggester