You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/03 20:24:28 UTC
[GitHub] [airflow] adambaker opened a new issue, #28704: EMR Add Steps Op succeeds at adding steps but fails.
adambaker opened a new issue, #28704:
URL: https://github.com/apache/airflow/issues/28704
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
6.2.0
### Apache Airflow version
2.5.0
### Operating System
CentOS 7
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### What happened
Upon upgrading to airflow 2.5.0, EMR Add Steps Operator now:
1. Succeeds at adding a step to our emr job.
2. tries to wait for the job to complete
3. times out, retries, and duplicates the steps on our emr cluster.
4. times out, and fails.
The cause seems to be an error in the operator:
https://github.com/apache/airflow/pull/27286/files#diff-94612f7f88a5f95ca9e3f23051f39f1e0b6a4cadac1f533615db5480b379ed34R115
Instead of passing `wait_for_completion=True` to the emr hook method `add_job_flow_steps`, it should probably pass `self.wait_for_completion`. Also, currently `self.wait_for_completion` seem to not be used anywhere in the project.
### What you think should happen instead
The task should succeed once the EMR steps are added and move on to the next task in the dag.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] adambaker commented on issue #28704: EMR Add Steps Op succeeds at adding steps but fails.
Posted by GitBox <gi...@apache.org>.
adambaker commented on issue #28704:
URL: https://github.com/apache/airflow/issues/28704#issuecomment-1370194896
Looks like this is fixed on the main branch, so I guess we just need to wait for it to be built in to the next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28704: EMR Add Steps Op succeeds at adding steps but fails.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28704:
URL: https://github.com/apache/airflow/issues/28704#issuecomment-1370191900
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] adambaker closed issue #28704: EMR Add Steps Op succeeds at adding steps but fails.
Posted by GitBox <gi...@apache.org>.
adambaker closed issue #28704: EMR Add Steps Op succeeds at adding steps but fails.
URL: https://github.com/apache/airflow/issues/28704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org