You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kacpermuda (via GitHub)" <gi...@apache.org> on 2025/04/08 11:36:25 UTC
[I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
kacpermuda opened a new issue, #48928:
URL: https://github.com/apache/airflow/issues/48928
### Apache Airflow version
main (development)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
While the task was running I've set it's state to success from the UI. It caused the task to fail with the following error: `RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.0`. Full logs attached below.
### What you think should happen instead?
I think the task should be marked as success and no error should be raised.
### How to reproduce
1. Run the below DAG
2. While the task is running, set it's state from the UI:
```
import datetime as dt
from airflow import DAG
from airflow.providers.standard.operators.bash import BashOperator
with DAG(
dag_id="dag_success",
start_date=dt.datetime(2024, 7, 3),
schedule=None,
) as dag:
task_success = BashOperator(
task_id="task_success",
bash_command="sleep 10 & exit 0;",
)
```
### Operating System
MacOS
### Versions of Apache Airflow Providers
latest main
### Deployment
Virtualenv installation
### Deployment details
`breeze start-airflow -b postgres`
### Anything else?
Task logs
[task_logs.txt](https://github.com/user-attachments/files/19648195/task_logs.txt)
Scheduler logs (not sure if even needed)
[log_scheduler_20250408_112927.txt](https://github.com/user-attachments/files/19648178/log_scheduler_20250408_112927.txt)
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2788491010
@vatsrahul1001 can you try to reproduce this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2786934004
Even without the OL enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2788778487
@phanikumv @kacpermuda I also verified this, and I am getting the `SystemExit: -241` with `CeleryExecutor` when I make the task fail which is I think expected as we are terminating the process.
```
[2025-04-09, 08:26:15] ERROR - Task failed with exception: source="task"
SystemExit: -241
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 768 in run
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/task_runner.py", line 1031 in _execute_task
File "/opt/airflow/task-sdk/src/airflow/sdk/bases/operator.py", line 409 in wrapper
File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/bash.py", line 226 in execute
File "/opt/airflow/providers/standard/src/airflow/providers/standard/operators/bash.py", line 241 in _run_inline_command
File "/opt/airflow/providers/standard/src/airflow/providers/standard/hooks/subprocess.py", line 104 in run_command
File "/usr/local/lib/python3.9/site-packages/billiard/common.py", line 106 in _shutdown_cleanup
File "/usr/local/lib/python3.9/site-packages/billiard/pool.py", line 283 in exit
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2788746579
Checked again today with Celery, both with OL and without OL. In short, its always this error when changing state from UI while task is running (regardless whether we set success or failed state)(but for CeleryExecutor it's `SystemExit: -241` and not DB access error)
Used `breeze start-airflow -b postgres --executor CeleryExecutor` and `breeze start-airflow --integration openlineage -b postgres --executor=CeleryExecutor`. Here are the logs:
[task_celery_mark_success_no_ol.txt](https://github.com/user-attachments/files/19662994/task_celery_mark_success_no_ol.txt)
[task_celery_mark_failed_no_ol.txt](https://github.com/user-attachments/files/19662996/task_celery_mark_failed_no_ol.txt)
[task_celery_mark_success_with_ol.txt](https://github.com/user-attachments/files/19662993/task_celery_mark_success_with_ol.txt)
[task_celery_mark_failed_with_ol.txt](https://github.com/user-attachments/files/19662995/task_celery_mark_failed_with_ol.txt)
[log_celery_worker_with_ol.txt](https://github.com/user-attachments/files/19662997/log_celery_worker_with_ol.txt)
Verified again today on latest main on LocalExecutor as well (with and without OpenLineage) and it's still as described in the ticket: `RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.0`.
At least I couldn't reproduce the error from my message above (`[2025-04-08, 14:05:06] WARNING - Exception when importing 'airflow.providers.google.assets.gcs.sanitize_uri' from 'apache-airflow-providers-google' package: source="airflow.providers_manager"
RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.0`) that occurred when simply running the task, without any state modifications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #48928: [Airflow 3] Setting task's state from UI is causing db access error
URL: https://github.com/apache/airflow/issues/48928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2791169042
The following 3 PRs should fix this:
1. https://github.com/apache/airflow/pull/49025
2. https://github.com/apache/airflow/pull/49026
3. https://github.com/apache/airflow/pull/49023
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2788829464
@vatsrahul1001 It's also happening when marking task as success, and it's actually causing the task to go into failed state. Not sure if that's good 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2786932125
Does this only happen with OL enabled, or always?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2786583973
It also showed up in the logs when simply running task, source is the same I think:
<details>
```
[2025-04-08, 14:05:06] DEBUG - Plugins are already loaded. Skipping.: source="airflow.plugins_manager"
[2025-04-08, 14:05:06] DEBUG - Initialize hook lineage readers plugins: source="airflow.plugins_manager"
[2025-04-08, 14:05:06] DEBUG - Initializing Providers Manager[asset_uris]: source="airflow.providers_manager"
[2025-04-08, 14:05:06] WARNING - Exception when importing 'airflow.providers.google.assets.gcs.sanitize_uri' from 'apache-airflow-providers-google' package: source="airflow.providers_manager"
RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.0
File "/opt/airflow/airflow-core/src/airflow/providers_manager.py", line 308 in _correctness_check
File "/opt/airflow/airflow-core/src/airflow/utils/module_loading.py", line 39 in import_string
File "/usr/local/lib/python3.9/importlib/__init__.py", line 127 in import_module
File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850 in exec_module
File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
File "/opt/airflow/providers/google/src/airflow/providers/google/assets/gcs.py", line 21 in <module>
File "/opt/airflow/providers/google/src/airflow/providers/google/cloud/hooks/gcs.py", line 37 in <module>
File "/usr/local/lib/python3.9/site-packages/gcloud/aio/storage/__init__.py", line 1[56](http://localhost:28080/dags/dag_success/runs/manual__2025-04-08T14:04:54.573647+00:00/tasks/task_success?try_number=1#56) in <module>
File "/usr/local/lib/python3.9/site-packages/gcloud/aio/storage/blob.py", line 19 in <module>
File "/usr/local/lib/python3.9/site-packages/pyasn1/codec/der/decoder.py", line 9 in <module>
File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 846 in exec_module
File "<frozen importlib._bootstrap_external>", line 982 in get_code
File "<frozen importlib._bootstrap_external>", line 1039 in get_data
File "/opt/airflow/airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 227 in _exit_gracefully
File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line 101 in wrapper
File "/usr/local/lib/python3.9/contextlib.py", line 119 in __enter__
File "/opt/airflow/airflow-core/src/airflow/utils/session.py", line 41 in create_session
File "/opt/airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py", line 218 in __init__
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2788842595
@
> [@vatsrahul1001](https://github.com/vatsrahul1001) It's also happening when marking task as success, and it's actually causing the task to go into failed state. Not sure if that's good 😅
I am not able to replicate this when I try to change state to success. My task is getting succeeded only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2786927176
@kacpermuda Could you try this on CeleryExecutor and see if it happens there too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2789069535
Basically task-sdk kills it as task is not in running state anymore.
task_id='task_success', dag_id='dag_success', run_id='manual__2025-04-09T08:51:07.215955+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('dag_success.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=dag_success/run_id=manual__2025-04-09T08:51:07.215955+00:00/task_id=task_success/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-04-09 08:51:19.784064 [info ] Task finished [supervisor] duration=11.133604606991867 exit_code=0 final_state=success
2025-04-09 08:51:19.819381 [info ] Task execute_workload[1fd757d9-d78e-4f76-ae04-1f1017323d12] succeeded in 11.450090656988323s: None [celery.app.trace]
2025-04-09 09:17:24.500947 [info ] Task execute_workload[2dc6e25d-acd5-488a-b8fb-fa4190fb27b2] received [celery.worker.strategy]
2025-04-09 09:17:24.545338 [info ] [2dc6e25d-acd5-488a-b8fb-fa4190fb27b2] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('019619d7-0631-79c5-b9a0-8493fd7084e8'), task_id='task_success', dag_id='dag_success', run_id='manual__2025-04-09T09:17:24.393248+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('dag_success.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=dag_success/run_id=manual__2025-04-09T09:17:24.393248+00:00/task_id=task_success/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-04-09 09:17:39.961102 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_running', 'message': 'TI is no longer in the running state and task should terminate', 'current_state': 'success'}}
2025-04-09 09:17:39.961461 [error ] Server indicated the task shouldn't be running anymore [supervisor] detail={'detail': {'reason': 'not_running', 'message': 'TI is no longer in the running state and task should terminate', 'current_state': 'success'}} status_code=409 ti_id=UUID('019619d7-0631-79c5-b9a0-8493fd7084e8')
2025-04-09 09:17:44.968568 [info ] Process exited [supervisor] exit_code=0 pid=1112 signal=SIGTERM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #48928: [Airflow 3] Setting task's state from UI is causing db access error
URL: https://github.com/apache/airflow/issues/48928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "kacpermuda (via GitHub)" <gi...@apache.org>.
kacpermuda commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2786935388
I'll try CeleryExecutor first thing tomorrow and update the ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2789065149
Okay, after syncing with @kacpermuda, we found that if a task is marked as successful while it's still running, its state initially updates to 'success'. However, if you refresh the page multiple times, the task state changes to 'failed' with a -241 error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Airflow 3] Setting task's state from UI is causing db access error [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #48928:
URL: https://github.com/apache/airflow/issues/48928#issuecomment-2789289439
Ahhhh, so the issue is most likely after we get the error in the SDK of "TI is no longer in the running state and task should terminate", we shouldn't send the change state request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org