You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/20 06:31:36 UTC

[GitHub] [airflow] sha12br opened a new issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

sha12br opened a new issue #13785:
URL: https://github.com/apache/airflow/issues/13785


   Hi Team,
   
   Am trying to create a data pipeline where the EMR cluster gets provisioned, runs a spark job and terminate upon completion, so the flow would be similar as below
   
   EmrCreateJobFlowOperator --> EmrAddStepsOperator --> EmrTerminateJobFlowOperator
   
   The class "airflow.providers.amazon.aws.operators.emr_create_job_flow.EmrCreateJobFlowOperator" is taking in "region_name" as argument and it is easy to define the region in DAG script. Moreover this step works good and a cluster is provisioned.
   
   Where as the next step "EmrAddStepsOperator" --> the underlying class [airflow.providers.amazon.aws.operators.emr_add_steps.EmrAddStepsOperator] does not has any region_name as argument. So at first it throws up an error "ERROR - You must specify a region", since there is no such way to pass a region name to the class, i decided to create a new connection with conn_id as aws_shabr and connection uri as this --> aws://?region=eu-west-1   (so that it takes default login and pass, and region is mentioned). But upon running it still throws up the same error "ERROR - You must specify a region".
   
   ![image](https://user-images.githubusercontent.com/29857724/105135997-ce995580-5ae8-11eb-81d9-1bcedfb8d61e.png)
   
   link to Doc --> http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow-providers-amazon/latest/_api/airflow/providers/amazon/aws/operators/emr_add_steps/index.html
   
   Is there any other way to pass region_name to "EmrAddStepsOperator" step, if so please help me out here.
   
   Thanks in Advance
   Shabr


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13785:
URL: https://github.com/apache/airflow/issues/13785#issuecomment-763371326


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb closed issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

Posted by GitBox <gi...@apache.org>.
ashb closed issue #13785:
URL: https://github.com/apache/airflow/issues/13785


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13785:
URL: https://github.com/apache/airflow/issues/13785#issuecomment-774150220


   Check out https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html -- as dstandish mentioned, you can put the region_name in the extras (a bit easier than using the `config_kwargs`.)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #13785:
URL: https://github.com/apache/airflow/issues/13785#issuecomment-774181236


   nice i thought you could put it in extra but had trouble finding the spot in the code


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #13785:
URL: https://github.com/apache/airflow/issues/13785#issuecomment-773778390


   region is configurable in the airflow connection (and if using system creds, of course configurable on your system).  
   
   if using airflow connection to retrieve aws creds, you add kwargs for aws client by putting `config_kwargs` in conn extra: https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/base_aws.py#L371
   
   otherwise set region in system using either env vars or config files


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] dstandish commented on issue #13785: Apache Airflow's --> EmrAddStepsOperator throws up error as "ERROR - You must specify a region"

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #13785:
URL: https://github.com/apache/airflow/issues/13785#issuecomment-773778390


   region is configurable in the airflow connection (and if using system creds, of course configurable on your system).  
   
   if using airflow connection to retrieve aws creds, you add kwargs for aws client by putting `config_kwargs` in conn extra: https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/base_aws.py#L371
   
   otherwise set region in system using either env vars or config files


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org