You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "SameerMesiah97 (via GitHub)" <gi...@apache.org> on 2026/02/01 18:51:04 UTC
[PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
SameerMesiah97 opened a new pull request, #61333:
URL: https://github.com/apache/airflow/pull/61333
**Description**
Added best-effort cleanup for Redshift cluster creation to ensure clusters are deleted when failures occur after a cluster has been successfully created. Cleanup behavior is **guarded by a flag** and is **opted in by default**.
Previously, Redshift cluster creation could succeed via `create_cluster`, but the operator could then fail during post-creation steps when `wait_for_completion=True` and the IAM role lacking `redshift:DescribeClusters` permissions. In these cases, the Airflow task failed while the Redshift cluster continued provisioning or remained active in AWS, resulting in leaked infrastructure.
Cleanup has now been implemented for `RedshiftCreateClusterOperator`. If `WaiterError` is raised **after cluster creation has been initiated**. the operator attempts a best-effort deletion of the cluster. Cleanup failures are logged but do not mask or replace the original exception.
**Rationale**
Redshift cluster creation can succeed while post-creation steps fail. This commonly occurs with partially scoped IAM roles, for example, allowing `redshift:CreateCluster` but denying `redshift:DescribeClusters`, which is required by the availability waiter.
In these scenarios, the Airflow task fails while the cluster continues provisioning or running in AWS, leading to leaked infrastructure and ongoing cost. This change ensures that when a cluster has been started by the operator, failures during post-creation steps trigger a best-effort cleanup without altering error semantics or impacting unrelated resources.
**Tests**
* Added a unit test verifying that cluster deletion is attempted when a `WaiterError` occurs during the wait phase after successful cluster creation.
* Added a unit test ensuring that failures during cleanup do not mask or override the original exception raised by the waiter.
**Documentation**
The docstring for `RedshiftCreateClusterOperator` has been updated to document the new flag `delete_cluster_on_failure` and its default behavior.
**Backwards Compatibility**
A new flag called `delete_cluster_on_failure` has been added to `RedshiftCreateClusterOperator` with a default value of `True`. Best-effort cleanup will now be attempted if a post-creation failure (including `WaiterError`) occurs after the cluster has been successfully created.
Closes: #61324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #61333:
URL: https://github.com/apache/airflow/pull/61333#discussion_r2770585721
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/redshift_cluster.py:
##########
@@ -188,6 +192,7 @@ def __init__(
max_attempt: int = 5,
poll_interval: int = 60,
deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
+ delete_cluster_on_failure: bool = True,
Review Comment:
I'll leave the decision how to proceed to the AWS team @vincbeck @o-nikolas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #61333:
URL: https://github.com/apache/airflow/pull/61333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #61333:
URL: https://github.com/apache/airflow/pull/61333#issuecomment-3904987729
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #61333:
URL: https://github.com/apache/airflow/pull/61333#discussion_r2790535686
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/redshift_cluster.py:
##########
@@ -188,6 +192,7 @@ def __init__(
max_attempt: int = 5,
poll_interval: int = 60,
deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
+ delete_cluster_on_failure: bool = True,
Review Comment:
We've already had a few of these merged, and it's a nice feature for the operator to do this cleanup when most users won't even know to try to create something similar with task teardown logic. So I'm still in favour of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #61333:
URL: https://github.com/apache/airflow/pull/61333#discussion_r2753582483
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/redshift_cluster.py:
##########
@@ -188,6 +192,7 @@ def __init__(
max_attempt: int = 5,
poll_interval: int = 60,
deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
+ delete_cluster_on_failure: bool = True,
Review Comment:
> I don't think this is needed? Airflow has teardown feature https://airflow.apache.org/docs/apache-airflow/stable/howto/setup-and-teardown.html#setup-and-teardown
That is a perfectly valid point. But I believe this new flag handles clean-ups slightly differently than teardowns. It aims to prevent resource leaks as a result of a class of failures caused by partial IAM permissions, which users may not anticipate. Users can model cleanup with teardown tasks if they anticipate these failures, but that requires them to foresee fairly provider-specific failure modes. But then we are shifting the responsibility of handling unexpected resource leaks to DAG authors.
My understanding is that providers should handle this class of failure by default, and that’s what this flag is trying to do, without preventing users from using teardown for broader DAG-level lifecycle management.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] RedshiftCreateClusterOperator could leave clusters running after failure [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #61333:
URL: https://github.com/apache/airflow/pull/61333#discussion_r2752863336
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/redshift_cluster.py:
##########
@@ -188,6 +192,7 @@ def __init__(
max_attempt: int = 5,
poll_interval: int = 60,
deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
+ delete_cluster_on_failure: bool = True,
Review Comment:
I don't think this is needed?
Airflow has teardown feature
https://airflow.apache.org/docs/apache-airflow/stable/howto/setup-and-teardown.html#setup-and-teardown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org