拆分文本有千絲萬縷的短語

Question

我有一個包含一些有着千絲萬縷的單詞的列表，像

List<String> lookUp = new ArrayList<>(); 
lookUp.add("New York"); 
lookUp.add("Big Apple"); 

一個句子我想將它拆分成詞，但不要拆我的列表中給出的千絲萬縷的話。因此，一個例子

String sentence = "New York is also called Big Apple"; 

它應該返回我

["New York", "is", "also", "called", "Big Apple"] 

我開始寫一個算法，該算法首先由空格分割了一句，然後我做一個循環：對於每一個字我檢查這個詞，它在lookUp-list中是正確的鄰居，如果是true，則將這些單詞解析在一起。

1）想象一下，我的lookUp-list中還包含兩個以上單詞的不可分割的短語，比如「George W. Bush」 - >我的算法只會查找「George W.」和「W. Bush」，並且在查找列表中找不到它，所以它會將其分成3個字。

2）更重要的問題（您可以忽略問題1）：是否已經有庫或甚至是GATE插件（這樣我就不必重新發明輪子）？這對德語短語也存在嗎？我無法找到一個=（

Answer 1

另一種實現上Java 7不使用regular expressions：

List<String> lookUp = new ArrayList<>(); 
    lookUp.add("New York"); 
    lookUp.add("New Jersey"); 
    lookUp.add("Big Apple"); 
    lookUp.add("George W. Bush"); 

    String sentence = "New York is also called Big Apple . New Jersey is located near to New York . George W. Bush doesn't live in New Mexico`"; 

    String currentPhrase = ""; 
    List<String> parseResult = new ArrayList<>(); 

    for (String word : sentence.split("\\s+")) { 
     currentPhrase += (currentPhrase.isEmpty() ? "" : " ") + word; 
     if (lookUp.contains(currentPhrase)) { 
      parseResult.add(currentPhrase); 
      currentPhrase = ""; 
      continue; 
     } 
     boolean phraseFound = false; 
     for (String look : lookUp) 
      if (look.startsWith(currentPhrase)) { 
       phraseFound = true; 
       break; 
      } 

     if (!phraseFound) { 
      parseResult.addAll(Arrays.asList(currentPhrase.split("\\s+"))); 
      currentPhrase = ""; 
     } 
    } 

    System.out.println(parseResult); 

輸出是：

[New York, is, also, called, Big Apple, ., New Jersey, is, located, near, to, New York, ., George W. Bush, doesn't, live, in, New, Mexico]