EMR未能引導Airflow

我創建了一個bash來提供執行氣流的環境，但由於某種原因，腳本無法正常工作。EMR未能引導Airflow

如果我首先提供了EMR，然後執行腳本，它的工作很好。但是，如果我使用該腳本在自定義活動中執行不起作用。我試圖改變命令執行像sudo，但仍然無法正常工作。

#!/bin/bash 

# check for master node 
IS_MASTER=true 
if [ -f /mnt/var/lib/info/instance.json ] 
then 
    IS_MASTER=`cat /mnt/var/lib/info/instance.json | tr -d '\n ' | sed -n 's|.*\"isMaster\":\([^,]*\).*|\1|p'` 
fi 


if [ "$IS_MASTER" = "true}" ]; 
    then 
    # install mysql jdbc driver on sqoop 
    wget -qN -O ~/mysql-connector-java-5.1.39.tar.gz "https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.39.tar.gz" 
    tar -zxvf ~/mysql-connector-java-5.1.39.tar.gz && rm ~/mysql-connector-java-5.1.39.tar.gz 

    sudo mv ~/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar /usr/lib/sqoop/lib 

    sudo chmod 744 /usr/lib/sqoop/lib/mysql-connector-java-5.1.39-bin.jar 

    aws s3 cp s3://monet-datapipeline/scripts/emr_boostrap_scripts/airflow_boostrap ~/ --recursive --exclude "*.sh" 

    #create enviroment for airflow 
    virtualenv airflowenv -p python3 
    source ~/airflowenv/bin/activate 

    pip install --upgrade pip 
    pip install airflow 
    pip install boto3 

    airflow initdb 

    mv ~/carriola/airflow.cfg ~/airflow 

    airflow webserver -p 9030 

    airflow scheduler 
fi

這是代碼錯誤。

來自主節點的stderr。

mv: cannot stat â€˜/home/hadoop/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jarâ€™: No such file or directory 
chmod: cannot access â€˜/usr/lib/sqoop/lib/mysql-connector-java-5.1.39-bin.jarâ€™: No such file or directory 
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 25: /home/hadoop/airflowenv/bin/activate: No such file or directory 
You are using pip version 6.1.1, however version 8.1.2 is available. 
You should consider upgrading via the 'pip install --upgrade pip' command. 
     Exception: 
     Traceback (most recent call last): 
     File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main 
      status = self.run(options, args) 
     File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run 
      root=options.root_path, 
     File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 687, in install 
      requirement.uninstall(auto_confirm=True) 
     File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 730, in uninstall 
      paths_to_remove.remove(auto_confirm) 
     File "/usr/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 126, in remove 
      renames(path, new_path) 
     File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 292, in renames 
      shutil.move(old, new) 
     File "/usr/lib64/python2.7/shutil.py", line 303, in move 
      os.unlink(src) 
     OSError: [Errno 13] Permission denied: '/usr/bin/pip' 

You are using pip version 6.1.1, however version 8.1.2 is available. 
You should consider upgrading via the 'pip install --upgrade pip' command. 
    Command "/usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/mnt/tmp/pip-build-rmGy3J/sqlalchemy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-J6Ft9n-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /mnt/tmp/pip-build-rmGy3J/sqlalchemy 
You are using pip version 6.1.1, however version 8.1.2 is available. 
You should consider upgrading via the 'pip install --upgrade pip' command. 
    Exception: 
    Traceback (most recent call last): 
    File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 246, in main 
     status = self.run(options, args) 
    File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 352, in run 
     root=options.root_path, 
    File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 693, in install 
     **kwargs 
    File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 817, in install 
     self.move_wheel_files(self.source_dir, root=root) 
    File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 1018, in move_wheel_files 
     isolated=self.isolated, 
    File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 237, in move_wheel_files 
     clobber(source, lib_dir, True) 
    File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 208, in clobber 
     os.makedirs(destdir) 
    File "/usr/lib64/python2.7/os.py", line 157, in makedirs 
     mkdir(name, mode) 
    OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/site-packages/s3transfer' 

/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 31: airflow: command not found 
mv: cannot stat â€˜/home/hadoop/carriola/airflow.cfgâ€™: No such file or directory 
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 35: airflow: command not found 
/emr/instance-controller/lib/bootstrap-actions/1/airflow_bootstrap.sh: line 37: airflow: command not found

來源

2016-08-15 Carlos Eduardo

您可以使用：

cat /mnt/var/lib/info/instance.json | jq .isMaster

，而不是找isMaster真實與否。

來源

2017-01-31 21:56:29

EMR未能引導Airflow

回答

相關問題