You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/05/08 16:42:25 UTC
[PR] Fix flaky test_celery_integration: deterministic celery task registration [airflow]
potiuk opened a new pull request, #66602:
URL: https://github.com/apache/airflow/pull/66602
## Summary
`providers/celery/tests/integration/celery/test_celery_executor.py::TestCeleryExecutor::test_celery_integration` has been failing intermittently with `httpx.ConnectError: [Errno 111] Connection refused`. The test is supposed to short-circuit task execution with a fake `execute_workload`, but the worker was sometimes invoking the real one, which calls the Execution API and fails when no API server is running.
## Root cause
`_prepare_app` registers the fake task on a fresh `test_app` via:
```python
test_execute = test_app.task(name=execute_name)(execute)
```
`@app.task(...)` defaults to `shared=True`, which appends a `cons` callback to celery's **process-global** `celery._state._on_app_finalizers` set — for both the *real* `execute_workload` (decorated at module import) and the *fake* registered here.
When `start_worker(app=test_app)` calls `test_app.finalize()`, celery iterates that set and runs each `cons(test_app)`:
```python
def _announce_app_finalized(app):
callbacks = set(_on_app_finalizers)
for callback in callbacks:
callback(app)
```
Set iteration order is **hash-based and non-deterministic**. Each `cons` calls `test_app._task_from_fun(fn, name=\"execute_workload\")`, which short-circuits if the name is already in `_tasks`:
```python
if name not in self._tasks:
... create ...
else:
task = self._tasks[name] # keeps existing, doesn't replace
```
So whichever finalizer fires first wins. If the real one wins, the worker invokes the real `execute_workload` → `BaseExecutor.run_workload` → `supervise_task` → tries to connect to the Execution API at `localhost:8080` → `Connection refused`. The test asserts the `success` task ended in `State.SUCCESS`, gets `State.FAILED`. `@pytest.mark.flaky(reruns=5)` doesn't help because the cons-registration outcome is consistent within a process (function ids hash the same).
The trace from the failed runs shows the log line from `celery_executor_utils.py:221` (inside the *real* `execute_workload`), confirming the fake never won.
## Fix
Force the fake to win deterministically:
1. Finalize `test_app` explicitly. The real finalizer registers the real task in `test_app._tasks`.
2. `pop` that entry.
3. Register the fake via `test_app.task(name=..., shared=False)(execute)`. With `shared=False` the fake doesn't add itself to `_on_app_finalizers`, so it stays scoped to `test_app`. With the app already finalized, `lazy=True` short-circuits to eager registration, and the resulting `Task` object is what `mock.patch.object(celery_executor_utils, execute_name, test_execute)` patches in.
Also drop the `celery_executor_utils.execute_workload.__wrapped__ = execute` lines — celery dispatches tasks via `Task.run`, not `__wrapped__`, so those reassignments weren't redirecting anything.
## Tests
The change touches only the test setup. The existing `test_celery_integration` parametrizations are what verify the fix — they now reliably hit the fake `execute_workload` regardless of finalizer iteration order.
Reproduces the kind of failure seen in https://github.com/apache/airflow/actions/runs/25548565094/job/74993414446 (and similar recent flaky runs of this test).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.7)
Generated-by: Claude Code (Opus 4.7) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix flaky test_celery_integration: deterministic celery task registration [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66602:
URL: https://github.com/apache/airflow/pull/66602#issuecomment-4409795321
### Backport failed to create: v3-2-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/25579097738'> Run details </a>
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>❌</td>
<td>v3-2-test</td>
<td><a href="https://github.com/apache/airflow/commit/9575fb3bf9b9c636abb4619c2012029e3119a08d"><img src='https://img.shields.io/badge/Commit-9575fb3-red' alt='Commit Link'></a></td>
</tr>
</table>
You can attempt to backport this manually by running:
```bash
cherry_picker 9575fb3 v3-2-test
```
This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.
After you have resolved the conflicts, you can continue the backport process by running:
```bash
cherry_picker --continue
```
If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#how-to-backport-pr-with-cherry-picker-cli).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix flaky test_celery_integration: deterministic celery task registration [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #66602:
URL: https://github.com/apache/airflow/pull/66602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix flaky test_celery_integration: deterministic celery task registration [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #66602:
URL: https://github.com/apache/airflow/pull/66602#issuecomment-4409793263
Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to **[Airflow 3.2.2](https://github.com/apache/airflow/milestone/138)** based on: backport label targeting v3-2-test
If this milestone is not correct, please update it to the appropriate milestone.
> This comment was generated by [Milestone Tag Assistant](https://github.com/apache/airflow/blob/main/.github/workflows/milestone-tag-assistant.yml).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org