You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "thirtyseven (via GitHub)" <gi...@apache.org> on 2023/02/23 00:39:39 UTC
[GitHub] [airflow] thirtyseven opened a new issue, #29712: EMRHook.get_cluster_id_by_name() doesn't use pagination
thirtyseven opened a new issue, #29712:
URL: https://github.com/apache/airflow/issues/29712
### Apache Airflow version
2.5.1
### What happened
When using EMRHook.get_cluster_id_by_name or any any operator that depends on it (e.g. EMRAddStepsOperator), if the results of the ListClusters API call is paginated (e.g. if your account has more than 50 clusters in the current region), and the desired cluster is in the 2nd page of results, None will be returned instead of the cluster ID.
### What you think should happen instead
Boto's pagination API should be used and the cluster ID should be returned.
### How to reproduce
Use `EmrAddStepsOperator` with the `job_flow_name` parameter on an `aws_conn_id` with more than 50 EMR clusters in the current region.
### Operating System
Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==7.2.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nsAstro commented on issue #29712: EMRHook.get_cluster_id_by_name() doesn't use pagination
Posted by "nsAstro (via GitHub)" <gi...@apache.org>.
nsAstro commented on issue #29712:
URL: https://github.com/apache/airflow/issues/29712#issuecomment-1442425890
I took a stab at this but I am at a loss as to how to test without spinning up 51 EMR clusters myself. If anyone has a suggestion for how to whitebox, or test this, happy to contribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #29712: EMRHook.get_cluster_id_by_name() doesn't use pagination
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #29712:
URL: https://github.com/apache/airflow/issues/29712#issuecomment-1441070983
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #29712: EMRHook.get_cluster_id_by_name() doesn't use pagination
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #29712: EMRHook.get_cluster_id_by_name() doesn't use pagination
URL: https://github.com/apache/airflow/issues/29712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org