You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2025/04/11 16:33:36 UTC
[I] API server crashing with nested task group [airflow]
vatsrahul1001 opened a new issue, #49131:
URL: https://github.com/apache/airflow/issues/49131
### Apache Airflow version
3.0.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Getting error in API server and Grap is not visible with [example_task_group](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/example_dags/example_task_group.py)
<img width="1726" alt="Image" src="https://github.com/user-attachments/assets/1be5422e-2389-4ed2-ae81-ebbdb2d87666" />
```
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 187, in __call__
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 165, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 29, in __call__
await responder(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 126, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 46, in __call__
await self.app(scope, receive, self.send_with_compression)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 85, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 178, in __call__
recv_stream.close()
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 175, in __call__
response = await self.dispatch_func(request, call_next)
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/core_api/middleware.py", line 28, in dispatch
response = await call_next(request)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 153, in call_next
raise app_exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 140, in coro
await self.app(scope, receive_or_disconnect, send_no_error)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 714, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 73, in app
response = await f(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 301, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
return await run_in_threadpool(dependant.call, **values)
File "/usr/local/lib/python3.9/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
return await anyio.to_thread.run_sync(func)
File "/usr/local/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py", line 203, in grid_data
{
File "/opt/airflow/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py", line 207, in <dictcomp>
for (task_id_parent, run_id), tis in parent_tis.items()
RuntimeError: dictionary changed size during iteration
```
### What you think should happen instead?
_No response_
### How to reproduce
1. Run example DAG
2. Try to view grid view
### Operating System
Linux
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API server crashing with nested task group [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #49131:
URL: https://github.com/apache/airflow/issues/49131#issuecomment-2802442746
Thanks @vatsrahul1001 , I am able to reproduce it though confused on a plain shell updating a dictionary as it's iterated through dictionary comprehension doesn't produce the error though. I have created #49241 as a fix. While fixing this scenario I noticed new groups from the initial version won't have task instances for previous runs which leads to `IndexError` that is also handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API server crashing with nested task group [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #49131:
URL: https://github.com/apache/airflow/issues/49131#issuecomment-2798403570
I cannot reproduce this on main by triggering example_task_group and visiting the task instance pages. Is this consistently reproducible? Can you please share the Python version you are using?
http://localhost:8000/dags/example_task_group/runs/manual__2025-04-12T02:38:01.731507+00:00/tasks/section_1.task_2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API server crashing with nested task group [airflow]
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #49131:
URL: https://github.com/apache/airflow/issues/49131#issuecomment-2798398834
This happens when the dictionary is modified during iteration like below. The fix would be to wrap the `items()` call with a `list()` but given this seems to be a dictionary comprehension I am not sure why this occurs.
```python
data = {"a": 1}
for key, value in data.items():
data[key * 2] = value * 2
print(data)
```
```python
data = {"a": 1}
for key, value in list(data.items()):
data[key * 2] = value * 2
print(data)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API server crashing with nested task group [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb closed issue #49131: API server crashing with nested task group
URL: https://github.com/apache/airflow/issues/49131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API server crashing with nested task group [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49131:
URL: https://github.com/apache/airflow/issues/49131#issuecomment-2801616983
@tirkarthi I have repro steps
comment this out, do a run. Uncomment, do another run.
https://github.com/apache/airflow/blob/db436a1673479d59e5c2386b1e2f954a88658ab7/airflow-core/src/airflow/example_dags/example_task_group.py#L53-L58
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API server crashing with nested task group [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #49131:
URL: https://github.com/apache/airflow/issues/49131#issuecomment-2798513153
Yeah not able to reproduce it now. Looks like intermittent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org