You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/16 16:32:11 UTC

[GitHub] [airflow] rsotogar opened a new issue #17636: Invalid step_id argument in EmrStepSensor

rsotogar opened a new issue #17636:
URL: https://github.com/apache/airflow/issues/17636


   Hi!
   
   I am getting this error in the EmrStepSensor:
   
   `botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the DescribeStep operation: Step id '{ task_instance.xcom_pull(task_ids='users_pipeline', key='return_value')[0] }' is not valid`.
   
   Here's the code:
   
          `pipelines = ["events", "users", "eventsstaged", "activities", "feeds", "attendances", "friendships", "media", "downtoclown", "agents"]
   
           for pipeline in pipelines:
   
           watcher_operator = EmrStepSensor(task_id=f"{pipeline}_sensor",
                                            job_flow_id="{{ task_instance.xcom_pull(task_ids='start_emr_cluster', key='return_value')}}",
                                            step_id= f"{{ task_instance.xcom_pull(task_ids='{pipeline}_pipeline', key='return_value')[0] }}",
                                            target_states=["COMPLETED"])
   
           watcher_operators.append(watcher_operator)`
   
   I am using Spark 3.1.1 running on EMR 6.3.0 and Airflow is deployed on EC2 using Docker. When I use the XCOM method to extract the job flow id seems fine, it only gives me problems when extracting the step id. Any help would be appreciated.
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #17636: Invalid step_id argument in EmrStepSensor

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #17636:
URL: https://github.com/apache/airflow/issues/17636#issuecomment-900263204


   I'm closing as this isn't a bug but more of a support question.
   If further assistance is needed please use StackOverflow or troubleshooting channel in slack.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #17636: Invalid step_id argument in EmrStepSensor

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #17636:
URL: https://github.com/apache/airflow/issues/17636#issuecomment-900221900


   I think it may be just a wrong usage of Jinja with f string format.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #17636: Invalid step_id argument in EmrStepSensor

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #17636:
URL: https://github.com/apache/airflow/issues/17636


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #17636: Invalid step_id argument in EmrStepSensor

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17636:
URL: https://github.com/apache/airflow/issues/17636#issuecomment-899649288


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] uranusjr commented on issue #17636: Invalid step_id argument in EmrStepSensor

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17636:
URL: https://github.com/apache/airflow/issues/17636#issuecomment-900259662


   In an f-string, `{{` becomes `{`, and `}}` becomes `}`. So to get a valid Jinja template you need four open and four closing braces.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org