ElasticSearch引發日期字符串導入csv與Logstash時的轉換錯誤

我試圖運行一個簡單的csv文件與Logstash到ElasticSearch。ElasticSearch引發日期字符串導入csv與Logstash時的轉換錯誤

但是，當我運行它，我得到以下錯誤將字符串轉換爲日期格式（第一個日期列）。

"error"=>{ 
"type"=>"mapper_parsing_exception", 
"reason"=>"failed to parse [Date]", 
"caused_by"=>{ 
    "type"=>"illegal_argument_exception", 
    "reason"=>"Invalid format: \"Date\""}}}}

當我刪除日期列時，所有工作都很好。

我使用以下csv文件：

Date,Open,High,Low,Close,Volume,Adj Close 
2015-04-02,125.03,125.56,124.19,125.32,32120700,125.32 
2015-04-01,124.82,125.12,123.10,124.25,40359200,124.25 
2015-03-31,126.09,126.49,124.36,124.43,41852400,124.43 
2015-03-30,124.05,126.40,124.00,126.37,46906700,126.37

及以下logstash.conf：

input { 
    file { 
    path => "path/file.csv" 
    type => "core2" 
    start_position => "beginning"  
    } 
} 
filter { 
    csv { 
     separator => "," 
     columns => ["Date","Open","High","Low","Close","Volume","Adj Close"] 
    } 
    mutate {convert => ["High", "float"]} 
    mutate {convert => ["Open", "float"]} 
    mutate {convert => ["Low", "float"]} 
    mutate {convert => ["Close", "float"]} 
    mutate {convert => ["Volume", "float"]} 
    date { 
    match => ["Date", "yyyy-MM-dd"] 
    target => "Date" 
    } 
} 
output { 
    elasticsearch { 
     action => "index" 
     hosts => "localhost" 
     index => "stock15" 
     workers => 1 
    } 
    stdout {} 
}

看來我處理日期罰款。任何想法可能會出錯？

謝謝！

來源

2016-12-25 Guy Dubrovski

感謝@Yeikel，我結束了改變logstash配置，而不是數據本身。

在應用csv過濾器之前，我使用正則表達式來檢查它是否是標題。所以，如果是標頭，我放棄它，並繼續到下一行（將要與CSV過濾器處理）

請參閱更新配置，解決了頭問題：

input { 
    file { 
    path => "path/file.csv" 
    start_position => "beginning"  
    } 
} 
filter { 
    if ([message] =~ "\bDate\b") { 
     drop { } 
    } else { 
     csv { 
      separator => "," 
      columns => ["Date","Open","High","Low","Close","Volume","Adj Close"] 
     } 
     mutate {convert => ["High", "float"]} 
     mutate {convert => ["Open", "float"]} 
     mutate {convert => ["Low", "float"]} 
     mutate {convert => ["Close", "float"]} 
     mutate {convert => ["Volume", "float"]} 
     date { 
     match => ["Date", "yyyy-MM-dd"] 
     } 
    } 
} 
output { 
    elasticsearch { 
     action => "index" 
     hosts => "localhost" 
     index => "stock15" 
     workers => 1 
    } 
    stdout { 
     codec => rubydebug 
    } 
}

來源

2016-12-27 08:00:28

問題出在文件本身。 Logstash正在讀的第一線，這是無法解析：

Date,Open,High,Low,Close,Volume,Adj Close

權不是解決它刪除該文件的標題：

2015-04-02,125.03,125.56,124.19,125.32,32120700,125.32 
2015-04-01,124.82,125.12,123.10,124.25,40359200,124.25 
2015-03-31,126.09,126.49,124.36,124.43,41852400,124.43 
2015-03-30,124.05,126.40,124.00,126.37,46906700,126.37

它應該沒問題。

有關於這在GitHub上的問題：https://github.com/elastic/logstash/issues/2088

來源

2016-12-26 22:40:31 Yeikel

ElasticSearch引發日期字符串導入csv與Logstash時的轉換錯誤

回答

相關問題