8
我注意到mrjob和boto都不支持Python界面在Amazon Elastic MapReduce(EMR)上提交和運行Hive作業。是否有任何其他支持在EMR上運行Hive的Python客戶端庫?Python客戶端支持在Amazon EMR之上運行Hive
我注意到mrjob和boto都不支持Python界面在Amazon Elastic MapReduce(EMR)上提交和運行Hive作業。是否有任何其他支持在EMR上運行Hive的Python客戶端庫?Python客戶端支持在Amazon EMR之上運行Hive
隨着博託你可以做這樣的事情:
args1 = [u's3://us-east-1.elasticmapreduce/libs/hive/hive-script',
u'--base-path',
u's3://us-east-1.elasticmapreduce/libs/hive/',
u'--install-hive',
u'--hive-versions',
u'0.7']
args2 = [u's3://us-east-1.elasticmapreduce/libs/hive/hive-script',
u'--base-path',
u's3://us-east-1.elasticmapreduce/libs/hive/',
u'--hive-versions',
u'0.7',
u'--run-hive-script',
u'--args',
u'-f',
s3_query_file_uri]
steps = []
for name, args in zip(('Setup Hive','Run Hive Script'),(args1,args2)):
step = JarStep(name,
's3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar',
step_args=args,
#action_on_failure="CANCEL_AND_WAIT"
)
#should be inside loop
steps.append(step)
# Kick off the job
jobid = EmrConnection().run_jobflow(name, s3_log_uri,
steps=steps,
master_instance_type=master_instance_type,
slave_instance_type=slave_instance_type,
num_instances=num_instances,
hadoop_version="0.20")
奏效 - 感謝unthingable! – poiuy 2011-07-29 23:17:19
由於VALIDATION_ERROR,我正在終止emr ..任何想法? – vks 2017-06-19 13:47:13