[jira] [Created] (AIRFLOW-1936) EmrCreateJobFlowOperator is unable to launch the aws emr cluster

["Vikram Fugro (JIRA)" <ji...@apache.org>]

Vikram Fugro created AIRFLOW-1936:
-------------------------------------

             Summary: EmrCreateJobFlowOperator is unable to launch the aws emr cluster
                 Key: AIRFLOW-1936
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1936
             Project: Apache Airflow
          Issue Type: Bug
          Components: aws, boto3, DAG, hooks, operators
    Affects Versions: Airflow 1.8
         Environment: ubuntu 16.04 , python 2.7, boto3
            Reporter: Vikram Fugro
         Attachments: dag_logs.txt, emr_hook_with_hardcoded_values.py, my_emr_boto_script(works).py, my_emr_dag(doesNotWork).py

The EmrCreateJobFlowOperator  operator is unable to create the emr cluster, although it returns  success. The most strange thing is that I see that it in logs, it also returns the jobflow-Id but I see no cluster starting up in the aws emr console. If I try to launch the emr with my boto3 script , it works. I even  hardcoded the arguments (taken from my boto3 script as is)  in self.get_conn().run_job_flow(), in contrib/hooks/emr_hook.py , but no luck. 

I am running airflow 1.8, localexecutor with metadb postgres. I have tried both ways ; setting up a schedule and also by  command 'airflow run  -f emr_job_flow_manual_steps_dag2 create_job_flow 2017-12-18' 

I am attaching the logs my dag,  my boto3 script,  emr_hook.py(changed only for testing purpose)  and  my dag.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)