如何爲斯坦福大學關係抽取生成自定義培訓數據

Question

我已經培訓了一個自定義分類器來理解金融領域中的命名實體。我想要生成如下所示的自定義培訓數據鏈接 http://cogcomp.cs.illinois.edu/Data/ER/conll04.corp

我可以手工標記自定義關係，但想要先用我的自定義命名實體生成conll數據格式。

我也嘗試過以下方式的解析器，但不會生成關係訓練數據，如鏈接https://nlp.stanford.edu/software/relationExtractor.html#training中提到的Roth和Yih的數據。

的Java -mx150m -cp 「斯坦福解析器 - 全2013年6月20日/ *：」 edu.stanford.nlp.parser.lexparser.LexicalizedParser -outputFormat 「佩恩」 EDU /斯坦福/ NLP /模型/ lexparser /englishPCFG.ser.gz stanford-parser-full-2013-06-20/data/testsent.txt> testsent.tree

java -mx150m -cp「stanford-parser-full-2013-06-20/*/：」 edu.stanford.nlp.trees.EnglishGrammaticalStructure -treeFile testsent.tree -conllx

以下是定製NER運行的輸出與下面的Python代碼

'java -mx2g -cp "*" edu.stanford.nlp.ie.NERClassifierCombiner '\ 
       '-ner.model classifiers\custom-model.ser.gz '\ 
       'classifiers/english.all.3class.distsim.crf.ser.gz,'\ 
       'classifiers/english.conll.4class.distsim.crf.ser.gz,'\ 
       'classifiers/english.muc.7class.distsim.crf.ser.gz ' \ 
       '-textFile '+ outtxt_sent + ' -outputFormat inlineXML > ' + outtxt + '.ner' 

output: 

<PERSON>Charles Sinclair</PERSON> <DESG>Chairman</DESG> <ORGANIZATION>-LRB- age 68 -RRB- Charles was appointed a</ORGANIZATION> <DESG>non-executive director</DESG> <ORGANIZATION>in</ORGANIZATION> 

分開

所以NER獨立工作，即使我有java代碼來測試它。

這裏是關係數據生成詳細的代碼

Properties props = new Properties(); 
     props.setProperty("annotators", "tokenize,ssplit,pos,lemma,ner,entitymentions"); 
     props.setProperty("ner.model", "classifiers/custom-model.ser.gz,classifiers/english.all.3class.distsim.crf.ser.gz,classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz"); 
     // set up Stanford CoreNLP pipeline 
     StanfordCoreNLP pipeline = new StanfordCoreNLP(props); 
     // build annotation for a review 
     Annotation annotation = new Annotation("Charles Sinclair Chairman -LRB- age 68 -RRB- Charles was appointed a non-executive director"); 
     pipeline.annotate(annotation); 
     int sentNum = 0; 

.............. Rest of the code is same as yours 

output: 
0 PERSON 0 O NNP/NNP Charles/Sinclair O O O 
0 PERSON 1 O NNP Chairman O O O 
0 PERSON 2 O -LRB-/NN/CD/-RRB-/NNP/VBD/VBN/DT -LRB-/age/68/-RRB-/Charles/was/appointed/a O O O 
0 PERSON 3 O JJ/NN non-executive/director O O O 

O 3 member_of_board //I will modify the relation once the data generated with proper NER 

The Ner tagging is ok now. 
props.setProperty("ner.model", "classifiers/classifiers/english.all.3class.distsim.crf.ser.gz,classifiers/english.conll.4class.distsim.crf.ser.gz,classifiers/english.muc.7class.distsim.crf.ser.gz,"); 

定製NER問題就迎刃而解了。

Answer 1

此鏈接顯示的數據的一個例子：http://cogcomp.cs.illinois.edu/Data/ER/conll04.corp

我不認爲這是在斯坦福CoreNLP產生此問題的方法。

標記數據後，您需要遍歷句子並以相同格式打印出令牌，包括詞性標記和ner標記。看來大部分列中都有「O」。

對於每個有關係的句子，您需要在關係格式中的句子後面打印出一行。例如，此行表示前面的句子具有Live_In關係：

7 0 Live_In 

以下是生成句子輸出的一些示例代碼。您需要將管道設置爲使用您的ner模型，而不是將ner.model屬性設置爲您的自定義模型的路徑。警告：此代碼中可能存在一些錯誤，但它應該顯示如何從StanfordCoreNLP數據結構訪問您需要的數據。

package edu.stanford.nlp.examples; 

import edu.stanford.nlp.ling.*; 
import edu.stanford.nlp.pipeline.*; 
import edu.stanford.nlp.trees.*; 
import edu.stanford.nlp.util.*; 

import java.util.*; 
import java.util.stream.Collectors; 

public class CreateRelationData { 

    public static void main(String[] args) { 
    // set up pipeline properties 
    Properties props = new Properties(); 
    props.setProperty("annotators", "tokenize,ssplit,pos,lemma,ner,entitymentions"); 
    // set up Stanford CoreNLP pipeline 
    StanfordCoreNLP pipeline = new StanfordCoreNLP(props); 
    // build annotation for a review 
    Annotation annotation = new Annotation("Joe Smith lives in Hawaii."); 
    pipeline.annotate(annotation); 
    int sentNum = 0; 
    for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) { 
     int tokenNum = 1; 
     int elementNum = 0; 
     int entityNum = 0; 
     CoreMap currEntityMention = sentence.get(CoreAnnotations.MentionsAnnotation.class).get(entityNum); 
     String currEntityMentionWords = currEntityMention.get(CoreAnnotations.TokensAnnotation.class).stream().map(token -> token.word()). 
      collect(Collectors.joining("/")); 
     String currEntityMentionTags = 
      currEntityMention.get(CoreAnnotations.TokensAnnotation.class).stream().map(token -> token.tag()). 
       collect(Collectors.joining("/")); 
     String currEntityMentionNER = currEntityMention.get(CoreAnnotations.EntityTypeAnnotation.class); 
     while (tokenNum <= sentence.get(CoreAnnotations.TokensAnnotation.class).size()) { 
     if (currEntityMention.get(CoreAnnotations.TokensAnnotation.class).get(0).index() == tokenNum) { 
      String entityText = currEntityMention.toString(); 
      System.out.println(sentNum+"\t"+currEntityMentionNER+"\t"+elementNum+"\t"+"O\t"+currEntityMentionTags+"\t"+ 
       currEntityMentionWords+"\t"+"O\tO\tO"); 
      // update tokenNum 
      tokenNum += (currEntityMention.get(CoreAnnotations.TokensAnnotation.class).size()); 
      // update entity if there are remaining entities 
      entityNum++; 
      if (entityNum < sentence.get(CoreAnnotations.MentionsAnnotation.class).size()) { 
      currEntityMention = sentence.get(CoreAnnotations.MentionsAnnotation.class).get(entityNum); 
      currEntityMentionWords = currEntityMention.get(CoreAnnotations.TokensAnnotation.class).stream().map(token -> token.word()). 
       collect(Collectors.joining("/")); 
      currEntityMentionTags = 
       currEntityMention.get(CoreAnnotations.TokensAnnotation.class).stream().map(token -> token.tag()). 
        collect(Collectors.joining("/")); 
      currEntityMentionNER = currEntityMention.get(CoreAnnotations.EntityTypeAnnotation.class); 
      } 
     } else { 
      CoreLabel token = sentence.get(CoreAnnotations.TokensAnnotation.class).get(tokenNum-1); 
      System.out.println(sentNum+"\t"+token.ner()+"\t"+elementNum+"\tO\t"+token.tag()+"\t"+token.word()+"\t"+"O\tO\tO"); 
      tokenNum += 1; 
     } 
     elementNum += 1; 
     } 
     sentNum++; 
    } 
    System.out.println(); 
    System.out.println("O\t3\tLive_In"); 
    } 
}