從Logstash加載JSON流到Elasticsearch

我想通過logstash加載JSON流（json對象文件的行）到elasticsearch。我的一些json對象字段包含unicode，如下所示。從Logstash加載JSON流到Elasticsearch

{"status_link": "https://www.facebook.com/asia/videos/1118055131588324/", "num_loves": "4", "num_sads": "0", "num_wows": "0", "num_angrys": "0", "num_comments": "6", "num_reactions": "46", "num_hahas": "0", "link_name": "", "num_likes": "42", "timestamp": "2016-07-25 02:07:38", "num_shares": "8", "_id": "156915824368931_1118055131588324", "status_message": "\"\u0411\u0440\u0438\u0433\u0430\u0434\" \u0440\u0435\u0430\u043b\u0438\u0442\u0438 \u0448\u043e\u0443\u043d\u044b \u0448\u0438\u043d\u044d \u0434\u0443\u0433\u0430\u0430\u0440 07-\u0440 \u0441\u0430\u0440\u044b\u043d 28-\u043d\u044b \u043f\u04af\u0440\u044d\u0432 \u0433\u0430\u0440\u0430\u0433\u0438\u0439\u043d \u043e\u0440\u043e\u0439 18:00 \u0446\u0430\u0433\u0430\u0430\u0441", "status_type": "video"}

當我開始logstash，它給了我一個錯誤：

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Mixing up field types: class org.elasticsearch.index.mapper.core.StringFieldMapper$StringFieldType != class org.elasticsearch.index.mapper.internal.IdFieldMapper$IdFieldType on field _id"}}}}, :level=>:warn}

我logstash.conf：

input 
{ 
    file 
    { 
     path => "test.json" 
     start_position => "beginning" 
     sincedb_path => "/dev/null" 
     exclude => "*.gz" 
     type => "posts" 
     codec => "json" 
    } 
} 

filter { 
    json { 
    source => "message" 
    } 
} 

output { 
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "fb" 
    codec => "json" 
    } 
}

我試圖加載JSON對象沒有統一代碼，它成功地解析和elasticsearch中的索引。

來源

2016-08-04 KevinOelen

問題是您的文檔中有_id字段。 _id是一個保存字段。所以你需要刪除或重命名它。

來源

2016-08-04 11:27:41 alpert

稍後，我正在加載評論，並將其與帖子進行父子關係。那麼是不是隻有_id字段才能在帖子和評論之間建立聯繫？ – KevinOelen

如果您想使用'_id'作爲您的文檔ID，我的建議將其重命名爲id並在您的elasticsearch輸出中使用'document_id =>「％{[event] [id]}」'''。 – alpert

從Logstash加載JSON流到Elasticsearch

回答

相關問題