AWS Data Pipeline - 在創建EMR期間設置Hive站點值

我們正在將我們的數據管道版本從3.3.2升級到5.8，因此舊AMI版本上的引導操作已更改爲使用configuration進行設置，並在分類/屬性定義。AWS Data Pipeline - 在創建EMR期間設置Hive站點值

所以我的Json看起來像下面

{ 
      "enableDebugging": "true", 
      "taskInstanceBidPrice": "1", 
      "terminateAfter": "2 Hours", 
      "name": "ExportCluster", 
      "taskInstanceType": "m1.xlarge", 
      "schedule": { 
       "ref": "Default" 
      }, 
      "emrLogUri": "s3://emr-script-logs/", 
      "coreInstanceType": "m1.xlarge", 
      "coreInstanceCount": "1", 
      "taskInstanceCount": "4", 
      "masterInstanceType": "m3.xlarge", 
      "keyPair": "XXXX", 
      "applications": ["hadoop","hive", "tez"], 
      "subnetId": "XXXXX", 
      "logUri": "s3://pipelinedata/XXX", 
      "releaseLabel": "emr-5.8.0", 
      "type": "EmrCluster", 
      "id": "EmrClusterWithNewEMRVersion", 
      "configuration": [ 
       { "ref": "configureEmrHiveSite" } 
      ] 
     }, 
     { 
      "myComment": "This object configures hive-site xml.", 
      "name": "HiveSite Configuration", 
      "type": "HiveSiteConfiguration", 
      "id": "configureEmrHiveSite", 
      "classification": "hive-site", 
      "property": [ 
       {"ref": "hive-exec-compress-output" } 
      ] 
     }, 
     { 
      "myComment": "This object sets a hive-site configuration 
      property value.", 
      "name":"hive-exec-compress-output", 
      "type": "Property", 
      "id": "hive-exec-compress-output", 
      "key": "hive.exec.compress.output", 
      "value": "true" 
     } 
    ], 
    "parameters": []

通過上述JSON文件就被加載到數據管道，但拋出一個錯誤說

Object:HiveSite Configuration 
ERROR: 'HiveSiteConfiguration' 
Object:ExportCluster 
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'

我不知道這到底是什麼意思，可能你請讓我知道如果我正確地指定這個我認爲我是根據http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html

來源

2017-08-25 Krish

您能否成功升級到5.x？我特別對此步驟有一個疑問，而不更改默認配置。 https://stackoverflow.com/questions/47858108/how-to-upgrade-data-pipeline-definition-from-emr-3-x-to-4-x-5-x – user1322092

下面的塊應該有名稱爲「EMR配置配給「，只有通過AWS Data管道正確識別它，Hive-site.xml纔會相應設置。

{ 
     "myComment": "This object configures hive-site xml.", 
     "name": "EMR Configuration", 
     "type": "EmrConfiguration", 
     "id": "configureEmrHiveSite", 
     "classification": "hive-site", 
     "property": [ 
      {"ref": "hive-exec-compress-output" } 
     ] 
    },

來源

2017-08-28 15:26:59 Krish

AWS Data Pipeline - 在創建EMR期間設置Hive站點值

回答

相關問題