將Google API JSON文件導入Elasticsearch

Question

對於ELK堆棧，尤其是ES，我是全新的。 我正在嘗試導入使用Google Admin SDK API的JSON文件，並且想將其導入Elasticsearch。

到目前爲止，這是我的數據的JSON結構：

{ 
"kind": "reports#activities", 
"nextPageToken": string, 
"items": [ 
{ 
"kind": "audit#activity", 
    "id": { 
    "time": datetime, 
    "uniqueQualifier": long, 
    "applicationName": string, 
    "customerId": string 
    }, 
    "actor": { 
    "callerType": string, 
    "email": string, 
    "profileId": long, 
    "key": string 
    }, 
    "ownerDomain": string, 
    "ipAddress": string, 
    "events": [ 
    { 
     "type": string, 
     "name": string, 
     "parameters": [ 
     { 
      "name": string, 
      "value": string, 
      "intValue": long, 
      "boolValue": boolean 
     } 
     ] 
    } 
    ] 
    } 
] 
} 

所以我決定先使用此命令上傳的JSON文件到ES：

curl -s -XPOST 'localhost:9200/_bulk' --data-binary @documents.json 

，但我得到了一些錯誤：

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [START_ARRAY]"},"status":400} 

我該怎麼辦？

謝謝你的幫助！

Answer 1

JSON似乎是定義了您的文檔結構，因此您首先需要創建一個索引並使用與該結構匹配的映射。在你的情況，你可以做這樣的：

curl -XPUT localhost:9200/reports -d '{ 
    "nextPageToken": { 
    "type": "string" 
    }, 
    "items": { 
    "properties": { 
     "kind": { 
     "type": "string" 
     }, 
     "id": { 
     "properties": { 
      "time": { 
      "type": "date", 
      "format": "date_time" 
      }, 
      "uniqueQualifier": { 
      "type": "long" 
      }, 
      "applicationName": { 
      "type": "string" 
      }, 
      "customerId": { 
      "type": "string" 
      } 
     } 
     }, 
     "actor": { 
     "properties": { 
      "callerType": { 
      "type": "string" 
      }, 
      "email": { 
      "type": "string" 
      }, 
      "profileId": { 
      "type": "long" 
      }, 
      "key": { 
      "type": "string" 
      } 
     } 
     }, 
     "ownerDomain": { 
     "type": "string" 
     }, 
     "ipAddress": { 
     "type": "string" 
     }, 
     "events": { 
     "properties": { 
      "type": { 
      "type": "string" 
      }, 
      "name": { 
      "type": "string" 
      }, 
      "parameters": { 
      "properties": { 
       "name": { 
       "type": "string" 
       }, 
       "value": { 
       "type": "string" 
       }, 
       "intValue": { 
       "type": "long" 
       }, 
       "boolValue": { 
       "type": "boolean" 
       } 
      } 
      } 
     } 
     } 
    } 
    } 
}' 

這個正在做，你可以遵循使用批量調用上面的結構，現在指數的reports#activities文件。批量調用的語法被精確地定義爲here，即您需要一個命令行（該怎麼做），然後是下一行文檔來源（要索引什麼），它不能包含任何新行！

所以，你需要像這樣重新格式化你的documents.json文件（確保在第二行之後添加一個新行）。另外請注意，我已經添加了一些虛擬數據來說明該過程：

{"index": {"_index": "reports", "_type": "activity"}} 
{"kind":"reports#activities","nextPageToken":"string","items":[{"kind":"audit#activity","id":{"time":"2016-05-31T00:00:00.000Z","uniqueQualifier":1,"applicationName":"string","customerId":"string"},"actor":{"callerType":"string","email":"string","profileId":1,"key":"string"},"ownerDomain":"string","ipAddress":"string","events":[{"type":"string","name":"string","parameters":[{"name":"string","value":"string","intValue":1,"boolValue":true}]}]}]}