You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "bahram-cdt (via GitHub)" <gi...@apache.org> on 2026/02/16 13:58:11 UTC
[PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
bahram-cdt opened a new pull request, #62016:
URL: https://github.com/apache/airflow/pull/62016
## What
Handle `CrawlerRunningException` in `GlueCrawlerOperator.execute()` instead of letting it fail the Airflow task.
## Why
When `start_crawler()` or `update_crawler()` is called while the crawler is already running (e.g., from a retry, overlapping DAG run, or boto3 internal retry after a timeout), the AWS Glue API raises `CrawlerRunningException`. Currently this propagates as an unhandled `ClientError`, causing the Airflow task to fail even though the crawler run completes successfully.
This is a common issue in production: the Glue console shows the crawler succeeded, but Airflow marks the task as failed and triggers alerts.
## What Changed
**`providers/amazon/src/airflow/providers/amazon/aws/operators/glue_crawler.py`**
- Wrapped `update_crawler()` with try/except: catches `CrawlerRunningException` and logs a warning (skips the update since the crawler is busy).
- Wrapped `start_crawler()` with try/except: catches `CrawlerRunningException` and logs a warning (waits for the existing run instead of failing).
- All other `ClientError` codes are re-raised as before.
- Added `from botocore.exceptions import ClientError` import.
**`providers/amazon/tests/unit/amazon/aws/operators/test_glue_crawler.py`**
- `test_execute_crawler_running_on_start`: verifies `CrawlerRunningException` on `start_crawler` is caught and the operator waits for the existing run.
- `test_execute_crawler_running_on_update`: verifies `CrawlerRunningException` on `update_crawler` is caught and `start_crawler` is still called.
- `test_execute_other_client_error_on_start_raises`: verifies non-`CrawlerRunningException` errors on `start_crawler` propagate.
- `test_execute_other_client_error_on_update_raises`: verifies non-`CrawlerRunningException` errors on `update_crawler` propagate.
## How to Test
```python
# Simulate CrawlerRunningException
from botocore.exceptions import ClientError
error = ClientError(
error_response={"Error": {"Code": "CrawlerRunningException", "Message": "Already running"}},
operation_name="StartCrawler",
)
# Previously: operator.execute() raises ClientError -> task fails
# Now: operator catches it, logs warning, waits for existing run -> task succeeds
```
---
^ Add meaningful description above
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{pr_number}.breaking.rst`, in `providers/amazon/newsfragments`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "bahram-cdt (via GitHub)" <gi...@apache.org>.
bahram-cdt commented on code in PR #62016:
URL: https://github.com/apache/airflow/pull/62016#discussion_r2813945059
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue_crawler.py:
##########
@@ -90,12 +100,37 @@ def execute(self, context: Context) -> str:
"""
crawler_name = self.config["Name"]
if self.hook.has_crawler(crawler_name):
- self.hook.update_crawler(**self.config)
+ try:
+ self.hook.update_crawler(**self.config)
+ except ClientError as e:
+ if e.response["Error"]["Code"] == "CrawlerRunningException":
+ if self.fail_on_already_running:
+ raise
+ self.log.warning(
+ "Crawler '%s' is currently running. "
+ "Skipping update and waiting for the existing run to complete.",
+ crawler_name,
+ )
+ else:
+ raise
else:
self.hook.create_crawler(**self.config)
self.log.info("Triggering AWS Glue Crawler")
- self.hook.start_crawler(crawler_name)
+ try:
+ self.hook.start_crawler(crawler_name)
+ except ClientError as e:
+ if e.response["Error"]["Code"] == "CrawlerRunningException":
Review Comment:
Thanks for the suggestion. Pushed the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3909478650
Please fix static checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3908641506
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "bahram-cdt (via GitHub)" <gi...@apache.org>.
bahram-cdt commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3910786105
@vincbeck Fixed ruff failure and pushed again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3915749613
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "bahram-cdt (via GitHub)" <gi...@apache.org>.
bahram-cdt commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3909687236
> It kinds of makes sense to receive this exception no? You're trying to update or a start a new job but it fails to do so because there is already one running. What do you think?
Good point! The exception is indeed telling us something meaningful — but I'd argue the correct response is to wait for the existing run, not to fail the task.
The most common cause in production is **retry-induced race conditions**: boto3's built-in retry fires `start_crawler()` a second time after a network timeout on the first (successful) call. The user didn't do anything wrong, and the crawler will complete successfully, but the Airflow task fails and triggers false alerts.
Since the operator already supports `wait_for_completion`, the natural behavior when a crawler is already running is to wait for it — the end state is identical to starting a fresh run and waiting.
For `update_crawler`, I agree the case is slightly weaker (we're skipping a config update), but the config rarely changes between runs, and the next successful run will pick it up. Failing the whole task seems disproportionate.
An alternative design: we could add a `fail_on_already_running: bool = False` parameter to make this opt-in, if the team prefers a non-breaking-default approach. Happy to adjust!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #62016:
URL: https://github.com/apache/airflow/pull/62016#discussion_r2813753190
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue_crawler.py:
##########
@@ -90,12 +100,37 @@ def execute(self, context: Context) -> str:
"""
crawler_name = self.config["Name"]
if self.hook.has_crawler(crawler_name):
- self.hook.update_crawler(**self.config)
+ try:
+ self.hook.update_crawler(**self.config)
+ except ClientError as e:
+ if e.response["Error"]["Code"] == "CrawlerRunningException":
+ if self.fail_on_already_running:
+ raise
+ self.log.warning(
+ "Crawler '%s' is currently running. "
+ "Skipping update and waiting for the existing run to complete.",
+ crawler_name,
+ )
+ else:
+ raise
else:
self.hook.create_crawler(**self.config)
self.log.info("Triggering AWS Glue Crawler")
- self.hook.start_crawler(crawler_name)
+ try:
+ self.hook.start_crawler(crawler_name)
+ except ClientError as e:
+ if e.response["Error"]["Code"] == "CrawlerRunningException":
Review Comment:
Same
##########
providers/amazon/src/airflow/providers/amazon/aws/operators/glue_crawler.py:
##########
@@ -90,12 +100,37 @@ def execute(self, context: Context) -> str:
"""
crawler_name = self.config["Name"]
if self.hook.has_crawler(crawler_name):
- self.hook.update_crawler(**self.config)
+ try:
+ self.hook.update_crawler(**self.config)
+ except ClientError as e:
+ if e.response["Error"]["Code"] == "CrawlerRunningException":
+ if self.fail_on_already_running:
+ raise
Review Comment:
```suggestion
if not self.fail_on_already_running or e.response["Error"]["Code"] == "CrawlerRunningException":
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3909772843
> An alternative design: we could add a fail_on_already_running: bool = False parameter to make this opt-in, if the team prefers a non-breaking-default approach. Happy to adjust!
I would personally prefer this solution so that users can decide which behavior they want
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "bahram-cdt (via GitHub)" <gi...@apache.org>.
bahram-cdt commented on PR #62016:
URL: https://github.com/apache/airflow/pull/62016#issuecomment-3910088229
> > An alternative design: we could add a fail_on_already_running: bool = False parameter to make this opt-in, if the team prefers a non-breaking-default approach. Happy to adjust!
>
> I would personally prefer this solution so that users can decide which behavior they want
Agreed Now:
- **`fail_on_already_running=True`** (default) — preserves current behavior, no change for existing users
- **`fail_on_already_running=False`** — opt-in: catches `CrawlerRunningException`, logs a warning, and waits for the existing run to complete
Added tests covering both modes. Let me know if you'd like any further adjustments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handle CrawlerRunningException gracefully in GlueCrawlerOperator [airflow]
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck merged PR #62016:
URL: https://github.com/apache/airflow/pull/62016
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org