如何在java中使用lucene添加自定義停用詞

Question

我正在使用lucene刪除英語停用詞，但我的要求是刪除英語停用詞和自定義停用詞。以下是我使用lucene刪除英文停用詞的代碼。

我的示例代碼：

public class Stopwords_remove { 
    public String removeStopWords(String string) throws IOException 
    { 
     StandardAnalyzer ana = new StandardAnalyzer(Version.LUCENE_30); 
     TokenStream tokenStream = new StandardTokenizer(Version.LUCENE_36,newStringReader(string)); 
     StringBuilder sb = new StringBuilder(); 
     tokenStream = new StopFilter(Version.LUCENE_36, tokenStream, ana.STOP_WORDS_SET); 
     CharTermAttribute token = tokenStream.getAttribute(CharTermAttribute.class); 
     while (tokenStream.incrementToken()) 
     { 
      if (sb.length() > 0) 
      { 
       sb.append(" "); 
      } 
      sb.append(token.toString()); 
     } 
     return sb.toString(); 
    } 

    public static void main(String args[]) throws IOException 
    { 
      String text = "this is a java project written by james."; 
      Stopwords_remove stopwords = new Stopwords_remove(); 
      stopwords.removeStopWords(text); 

    } 
} 

輸出：java project written james.

所需的輸出：java project james.

我怎樣才能做到這一點？

Answer 1

您可以將添加的停用詞添加到標準英語停用詞集的副本中，或者只添加到另一個StopFilter中。像：

TokenStream tokenStream = new StandardTokenizer(Version.LUCENE_36, new StringReader(string)); 
CharArraySet stopSet = CharArraySet.copy(Version.LUCENE_36, StandardAnalyzer.STOP_WORD_SET); 
stopSet.add("add"); 
stopSet.add("your"); 
stopSet.add("stop"); 
stopSet.add("words"); 
tokenStream = new StopFilter(Version.LUCENE_36, tokenStream, stopSet); 
//Or, if you just need the added stopwords in a standardanalyzer, you could just pass this stopfilter into the StandardAnalyzer... 
//analyzer = new StandardAnalyzer(Version.LUCENE_36, stopSet); 

或：

TokenStream tokenStream = new StandardTokenizer(Version.LUCENE_36, new StringReader(string)); 
tokenStream = new StopFilter(Version.LUCENE_36, tokenStream, StandardAnalyzer.STOP_WORDS_SET); 
List<String> stopWords = //your list of stop words..... 
tokenStream = new StopFilter(Version.LUCENE_36, tokenStream, StopFilter.makeStopSet(Version.LUCENE_36, stopWords)); 

如果你想創建自己的分析，你可能會得到更好的服務如下更像Analyzer documentation的示例的樣式。