You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Vikram Fugro (JIRA)" <ji...@apache.org> on 2017/12/18 10:59:00 UTC
[jira] [Updated] (AIRFLOW-1936) EmrCreateJobFlowOperator is unable
to launch the aws emr cluster
[ https://issues.apache.org/jira/browse/AIRFLOW-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vikram Fugro updated AIRFLOW-1936:
----------------------------------
Description:
The EmrCreateJobFlowOperator operator is unable to create the emr cluster, although it returns success. The most strange thing that I see that in the logs is that it also returns the jobflow-Id but I see no cluster starting up in the aws emr console. If I try to launch the emr with my boto3 script , it works. I even hardcoded the arguments (taken from my boto3 script as is) in self.get_conn().run_job_flow(), in contrib/hooks/emr_hook.py , but no luck.
I am running airflow 1.8, localexecutor with metadb postgres. I have tried both ways ; setting up a schedule and also by command 'airflow run -f emr_job_flow_manual_steps_dag2 create_job_flow 2017-12-18'
I am attaching the logs my dag, my boto3 script, emr_hook.py(changed only for testing purpose) and my dag.
was:
The EmrCreateJobFlowOperator operator is unable to create the emr cluster, although it returns success. The most strange thing is that I see that it in logs, it also returns the jobflow-Id but I see no cluster starting up in the aws emr console. If I try to launch the emr with my boto3 script , it works. I even hardcoded the arguments (taken from my boto3 script as is) in self.get_conn().run_job_flow(), in contrib/hooks/emr_hook.py , but no luck.
I am running airflow 1.8, localexecutor with metadb postgres. I have tried both ways ; setting up a schedule and also by command 'airflow run -f emr_job_flow_manual_steps_dag2 create_job_flow 2017-12-18'
I am attaching the logs my dag, my boto3 script, emr_hook.py(changed only for testing purpose) and my dag.
> EmrCreateJobFlowOperator is unable to launch the aws emr cluster
> ----------------------------------------------------------------
>
> Key: AIRFLOW-1936
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1936
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws, boto3, DAG, hooks, operators
> Affects Versions: Airflow 1.8
> Environment: ubuntu 16.04 , python 2.7, boto3
> Reporter: Vikram Fugro
> Labels: newbie
> Attachments: dag_logs.txt, emr_hook_with_hardcoded_values.py, my_emr_boto_script(works).py, my_emr_dag(doesNotWork).py
>
>
> The EmrCreateJobFlowOperator operator is unable to create the emr cluster, although it returns success. The most strange thing that I see that in the logs is that it also returns the jobflow-Id but I see no cluster starting up in the aws emr console. If I try to launch the emr with my boto3 script , it works. I even hardcoded the arguments (taken from my boto3 script as is) in self.get_conn().run_job_flow(), in contrib/hooks/emr_hook.py , but no luck.
> I am running airflow 1.8, localexecutor with metadb postgres. I have tried both ways ; setting up a schedule and also by command 'airflow run -f emr_job_flow_manual_steps_dag2 create_job_flow 2017-12-18'
> I am attaching the logs my dag, my boto3 script, emr_hook.py(changed only for testing purpose) and my dag.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)