Spring批處理 - 讀取多行日誌消息

我在使用spring集成配置的spring批處理應用程序中將多行日誌消息作爲單個消息讀取時遇到問題，此應用程序必須讀取多行日誌消息（示例異常堆棧跟蹤）作爲單個消息，稍後它必須處理和分類消息以進一步索引。每行都通過它的時間戳（上面提到的模式，即DATE_PATTERN）來標識，並且它可以繼續多行行，我試圖繼續讀取一條消息，直到通過重寫從SimpleRecordSeparatorPolicy中看到另一個時間戳爲止，當第二行到達preProcess方法I我爲isEndOfRecord返回true，但這不能按預期工作，任何人都可以幫助我通過識別時間戳模式來讀取提到的日誌文件嗎？Spring批處理 - 讀取多行日誌消息

我使用org.springframework.batch.item.file.FlatFileItemReader和org.springframework.batch.item.file.mapping.PassThroughLineMapper作爲映射器。

請參閱完成消息，

1）日誌消息文件：採樣消息test.log中

2013-10-19 07:05:32.253 [My First Class..] LOG LEVEl first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 first-message-line-1 
first-message-line-2 first-message-line-2 first-message-line-2 
first-message-line-3 first-message-line-3 first-message-line-3 
first-message-line-4 first-message-line-4 first-message-line-4 
first-message-line-5 first-message-line-5 
first-message-line-6 
2013-10-19 07:05:32.257 [My Second Class..] LOG LEVEl second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 second-message-line-1 
second-message-line-2 second-message-line-2 second-message-line-2 
second-message-line-3 second-message-line-3 second-message-line-3 
second-message-line-4 second-message-line-4 second-message-line-4 
second-message-line-5 second-message-line-5 
second-message-line-6 
2013-10-19 07:05:32.259 [My Third Class..] LOG LEVEl third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 third-message-line-1 
third-message-line-2 third-message-line-2 third-message-line-2 
third-message-line-3 third-message-line-3 third-message-line-3 
third-message-line-4 third-message-line-4 third-message-line-4 
third-message-line-5 third-message-line-5 
third-message-line-6

2）批量配置文件

<batch:job id="fileReadingJob"> 
     <batch:step id="flatFileReadingStep"> 
      <batch:tasklet > 
       <batch:chunk reader="reader" writer="writer" commit-interval="10" /> 
      </batch:tasklet> 
     </batch:step> 
    </batch:job> 

    <bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step"> 
     <property name="lineMapper"> 
      <bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper"/> 
     </property> 
     <property name="bufferedReaderFactory"> 
      <bean class="org.springframework.batch.item.file.DefaultBufferedReaderFactory"/> 
     </property> 
     <property name="recordSeparatorPolicy" > 
      <bean class="com.batchlog.explorer.batchio.FlatFileRecordSeperationPolicy"/> 
     </property> 
     <property name="resource" value="file:///#{systemProperties['logfolder']}/#{jobParameters['inputfile']}" /> 
    </bean> 
    <bean id="writer" class="com.batchlog.explorer.batchio.FlatFileWriter" scope="step"/> 
........

3）

public class FlatFileRecordSeperationPolicy extends SimpleRecordSeparatorPolicy { 

    public static final String STARTING_OF_THE_LINE = "-STARTING_OF_THE_LINE-"; 
    public static final String CONTINUATION_OF_THE_FILE = "-CONTINUATION_OF_THE_FILE-"; 
    public static final String END_OF_THE_LINE = "-END_OF_THE_LINE-"; 
    public static final String END_OF_THE_LINE_CHARACER = " \n "; 
    public static final String DATE_PATTERN ="^(?>\\d\\d){1,2}-(?:0?[1-9]|1[0-2])-(\\s)?(?:2[0123]|[01][0-9]):? (?:[0-5][0-9])(?::?(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?))?(?:Z|[+-](?:2[0123]|[01][0-9])(?::?(?:[0-5][0-9])))?.*?"; 


    @Override 
     public boolean isEndOfRecord(String line) { 
      if(line.matches(DATE_PATTERN) || line.startsWith(STARTING_OF_THE_LINE) 
          || line.contains(CONTINUATION_OF_THE_FILE) || line.startsWith(END_OF_THE_LINE)){ 
       if(isNextLineStarts(line) || line.startsWith(END_OF_THE_LINE)){ 
        return true;//to break line 
       } 
     } 
     return false; //to conitnue line 

    private boolean isNextLineStarts(String preProcessOfLine){ 
      if(preProcessOfLine.contains(CONTINUATION_OF_THE_FILE) && !preProcessOfLine.endsWith(CONTINUATION_OF_THE_FILE)){ 
       String[] lines = preProcessOfLine.split(CONTINUATION_OF_THE_FILE); 
       if(lines[1].trim().matches(DATE_PATTERN)){ 
        return true; 
       } 
      } 
      return false; 
    } 
    @Override 
     public String preProcess(String line) { 
      if(line.matches(DATE_PATTERN) && !line.contains(CONTINUATION_OF_THE_FILE)){ 
       line = new StringBuilder(STARTING_OF_THE_LINE).append(line).toString(); 
      }else if(line.startsWith(STARTING_OF_THE_LINE) && !line.contains(CONTINUATION_OF_THE_FILE)){ 
       line = new StringBuilder(line.substring(STARTING_OF_THE_LINE.length())).append(CONTINUATION_OF_THE_FILE).toString(); 
      }else if(line.contains(CONTINUATION_OF_THE_FILE) && !line.endsWith(CONTINUATION_OF_THE_FILE)){ 
       String[] lines = line.split(CONTINUATION_OF_THE_FILE); 
       if(lines[1].trim().matches(DATE_PATTERN)){ 
        line = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString();//.append(lines[1]).toString(); 
       }else{ 
        line = new StringBuilder(lines[0]).append(lines[1]).append(CONTINUATION_OF_THE_FILE).toString(); 
       } 
      } 
       return super.preProcess(line); 
    } 
    @Override 
     public String postProcess(String record) { 
      if(record.startsWith(END_OF_THE_LINE)){ 
       record = new StringBuilder(record.substring(END_OF_THE_LINE.length())).toString(); 
      }else if(record.contains(CONTINUATION_OF_THE_FILE) && !record.endsWith(CONTINUATION_OF_THE_FILE)){ 
       String[] lines = record.split(CONTINUATION_OF_THE_FILE); 
       if(lines[1].trim().matches(DATE_PATTERN)){ 
        record = new StringBuilder(END_OF_THE_LINE).append(lines[0]).toString(); 
       }else{ 
        record = new StringBuilder(lines[0]).append(lines[1]).toString(); 
       } 
      } 
      return super.postProcess(record); 
    }

來源

2013-10-26 Ashok Gudise

寫下你的自己的ItemReader，如multiorder-line example中所述或this post中所述。

來源

2013-10-28 07:45:16

嗨貝拉，感謝您的答覆，一些如何我結束時做同樣的事情，當我在GIT搜索。關注 Ashok G –

嗨Bellabax，感謝您的意見，我沒有花時間在這個到現在，因爲這是一個POC我在做我的休閒時間。按照你的建議，我寫了自己的ItemWriter＆Reader，它工作的很好。我在read Method中處理了所有的邏輯，並且爲了獲得更大的靈活性，我也實現了Chunk監聽器，以便在讀者中獲取上下文。感謝您的幫助。 –

您的問題並沒有轉入RecordSeparatorPolicy.isEndOfRecord（String）範例。 isEndOfRecored在內襯結尾放在最後一行時很好地工作。
例如，在DefaultRecordSeparatorPolicy中，它確保您有一個偶數爲的報價。最後的報價包含在所需記錄中。在你的情況下，你會過度閱讀一行。

使用後處理和預處理工作，可能你的基本理念，但你仍然從FlatFileItemReader得到FlatFileParseException的最後一行，當你到達EOL和readline返回null看到FlatFileItemReader applyRecordSeparatorPolicy（串線）。

private String applyRecordSeparatorPolicy(String line) throws IOException { 

     String record = line; 
     while (line != null && !recordSeparatorPolicy.isEndOfRecord(record)) { 
      line = this.reader.readLine(); 
      if (line == null) { 

       if (StringUtils.hasText(record)) { 
        // A record was partially complete since it hasn't ended but 
        // the line is null 
        throw new FlatFileParseException("Unexpected end of file before record complete", record, lineCount); 
       } 
       else { 
        // Record has no text but it might still be post processed 
        // to something (skipping preProcess since that was already 
        // done) 
        break; 
       } 
      } 
      else { 
       lineCount++; 
      } 
      record = recordSeparatorPolicy.preProcess(record) + line; 
     } 

     return recordSeparatorPolicy.postProcess(record); 

    }

在這種情況下，您的輸出文件將缺少基於commit-interval和isEndOfRecord邏輯的行。

所以基本上我建議使用不同的方法，bellabax解決方案爲你工作？

來源

2013-10-29 06:26:55

嗨海姆，對不起，遲到的反應，趕上其他一些優先事項，沒有得到機會再次工作。你說得對，實際上這種方法並不適合我（我的意思是isEndOfTheRecord），正如你所說的。我必須嘗試bellabax解決方案，我會盡快更新你。關注 Ashok Gudise –

Spring批處理 - 讀取多行日誌消息

回答

相關問題