You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rcrchawla (via GitHub)" <gi...@apache.org> on 2026/03/10 20:22:27 UTC
[D] Airflow task failed but spark kube app is running [airflow]
GitHub user rcrchawla created a discussion: Airflow task failed but spark kube app is running
### Body
Airflow task got failed where spark kube app is running. Although spark kube app is long running app most probably around 1-2 hour. And there are concurrently many task running at the same time usually it happens between 02:30 am - 03:45 am UTC.
Q) What causing issue ?
A) Airflow task failed while spark kube app running
Airflow version -- **3.0.4**
Setup config
2 API servers
2 workers
1 dag processor
2 schedulers
Deployment --> HELM Chart deployment on Azure Kubernetes
Please check below logs
Worker logs :
-------------------------------------
2026-03-10 02:33:56.191330 [info ] Task execute_workload[8cbabf91-009f-44a6-86d1-bef109c70341] succeeded in 2715.019189195242s: None [celery.app.trace]
2026-03-10 02:39:57.112078 [info ] Task finished [supervisor] duration=1723.7576029417105 exit_code=0 final_state=success
2026-03-10 02:39:57.128929 [info ] Task execute_workload[9b3f27ec-09b5-424e-8d5c-412e541f51e8] succeeded in 1723.8186896019615s: None [celery.app.trace]
2026-03-10 02:40:50.688403 [info ] Task finished [supervisor] duration=744.0669570546597 exit_code=0 final_state=success
2026-03-10 02:40:50.705538 [info ] Task execute_workload[b08ac31a-2ee7-4029-b897-753157b18475] succeeded in 744.139388079755s: None [celery.app.trace]
2026-03-10 02:42:11.649891 [info ] Task finished [supervisor] duration=756.7588595808484 exit_code=0 final_state=success
2026-03-10 02:42:11.666368 [info ] Task execute_workload[0351c271-194e-4e58-87e4-a9c224351ab1] succeeded in 756.8229349320754s: None [celery.app.trace]
2026-03-10 02:43:37.239128 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:38.119304 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:38.640468 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:39.247588 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:39.425843 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:39.618220 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:40.002999 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:40.582177 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:41.186771 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:41.510710 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:42.658853 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:43.171303 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:43.826966 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:44.330891 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:44.874859 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:44.922591 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:45.866775 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:46.194974 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:46.482845 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:46.750792 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:48.198838 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:48.462121 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:49.749467 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:50.029438 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:50.834835 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:51.334847 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:51.431052 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:51.537615 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:52.567197 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:52.967177 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:53.615078 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:54.513959 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:56.442819 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:57.527549 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:57.765172 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:57.982839 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:58.099625 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:58.534632 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:59.007106 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:43:59.947380 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
2026-03-10 02:44:02.200313 [warning ] Failed to send heartbeat. Will be retried [supervisor] failed_heartbeats=1 max_retries=3 ti_id=UUID('019cd54c-28b0-7e18-9a7b-71ba469bf545')
API Server
----------------------
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=152133 state=running ti_id=019cd542-0d3e-7467-9f7a-4dfc2d7f0017
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=155023 state=running ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-0.airflow-worker.de-services.svc.cluster.local current_pid=81402 state=running ti_id=019cd578-f8c1-7125-9906-ef64229dbba5
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=157917 state=running ti_id=019cd54c-28ad-7db0-b0f8-d64ed0916d78
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-0.airflow-worker.de-services.svc.cluster.local current_pid=86154 state=running ti_id=019cd542-0d45-75e4-95d5-a2c461e3e559
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd542-0d3e-7467-9f7a-4dfc2d7f0017
INFO: 10.10.12.52:40870 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO: 10.10.12.52:40880 - "GET /api/v2/version HTTP/1.1" 200 OK
2026-03-10 02:45:23 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=151395 ti_id=019cd542-0d47-7d93-a021-0cc2c9de7344
2026-03-10 02:45:23 [debug ] Refreshed token issued to Task [airflow.api_fastapi.execution_api.deps] refresh_when_less_than=120 valid_left=73
2026-03-10 02:45:23 [debug ] Refreshed token issued to Task [airflow.api_fastapi.execution_api.deps] refresh_when_less_than=120 valid_left=73
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd526-91bc-7461-8be3-aa7574c5f60b
2026-03-10 02:45:23 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=155023 ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
[2026-03-10T02:45:23.575+0000] {exceptions.py:77} ERROR - Error with id 9zBmdizJ
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 75, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 302, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 213, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cadwyn/structure/versions.py", line 474, in decorator
response = await self._convert_endpoint_response_to_version(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cadwyn/structure/versions.py", line 520, in _convert_endpoint_response_to_version
response_or_response_body: Union[FastapiResponse, object] = await run_in_threadpool(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
return await anyio.to_thread.run_sync(func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2476, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cadwyn/schema_generation.py", line 515, in __call__
return self._original_callable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/routes/xcoms.py", line 419, in set_xcom
session.flush()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3449, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3588, in _flush
with util.safe_reraise():
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3549, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
_emit_insert_statements(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
c = connection._execute_20(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 179, in execute
res = self._query(mogrified_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
db.query(q)
File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd54c-28ad-7db0-b0f8-d64ed0916d78
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=65618 state=running ti_id=019cd526-91bc-7461-8be3-aa7574c5f60b
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd526-91bc-7461-8be3-aa7574c5f60b
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=151858 state=running ti_id=019cd542-0d49-744c-aa72-a33d5ac4249d
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd542-0d45-75e4-95d5-a2c461e3e559
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd542-0d49-744c-aa72-a33d5ac4249d
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=152133 state=running ti_id=019cd542-0d3e-7467-9f7a-4dfc2d7f0017
2026-03-10 02:45:23 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=157917 state=running ti_id=019cd54c-28ad-7db0-b0f8-d64ed0916d78
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd542-0d3e-7467-9f7a-4dfc2d7f0017
What you think should happen instead?
Airflow task should run without getting failed.
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
GitHub link: https://github.com/apache/airflow/discussions/63298
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "hezeclark (via GitHub)" <gi...@apache.org>.
GitHub user hezeclark added a comment to the discussion: Airflow task failed but spark kube app is running
This is a common issue with Spark on Kubernetes in Airflow. The Airflow task marks as failed, but the Spark application continues running in the cluster. This happens because Spark on Kubernetes runs asynchronously by default.
Here are the root causes and fixes:
**Root cause 1: Using SparkSubmitOperator without proper failure handling**
The SparkSubmitOperator submits the job but may timeout or lose connection before the Spark app finishes:
**Root cause 2: KubernetesPodOperator without proper cleanup**
**Fix for orphaned Spark apps:**
Add a cleanup task that runs even when the main task fails:
Are you using SparkKubernetesOperator (from apache-airflow-providers-cncf-kubernetes), SparkSubmitOperator, or a custom operator?
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16113619
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh edited a comment on the discussion: Airflow task failed but spark kube app is running
@rcrchawla Yes give it a try and let me know if that fixes the issue. We can address performance if the error is resolved
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16109044
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
Yes give it a try and let me know if that fixes the issue. We can address performance if the error is resolved
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16109044
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Please find below airflow config :
[celery]
flower_url_prefix =
worker_concurrency = 16
[celery_kubernetes_executor]
kubernetes_queue = kubernetes
[core]
auth_manager = airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
colored_console_log = False
dags_folder = /opt/airflow/dags/repo/prod/dags
execution_api_server_url = http://airflow-api-server:8080/execution/
executor = CeleryExecutor
load_examples = False
remote_logging = False
[dag_processor]
bundle_refresh_check_interval = 43200
min_file_process_interval = 43200
print_stats_interval = 43200
stale_dag_threshold = 86400
[database]
sql_alchemy_engine_args = {"pool_pre_ping": true}
[scheduler]
run_duration = 41460
standalone_dag_processor = True
statsd_host = airflow-statsd
statsd_on = True
statsd_port = 9125
statsd_prefix = airflow
task_instance_heartbeat_timeout = 900
Thanks
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16073872
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Hi @shaealh ,
This is the only error I see in api server logs there is no such operational error.
One thing I notice in worker logs
this is the task
2026-03-10 02:14:46.785312 [info ] [0e6ea763-20c8-458c-bfe0-5660de056a59] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('019cd54c-28b0-7e18-9a7b-71ba469bf545'), task_id='stg_intl_user_useroptins.get_count', dag_id='staging-userservice', run_id='manual__2026-03-10T00:00:00+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=96, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('staging_userservice.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=staging-userservice/run_id=manual__2026-03-10T00:00:00+00:00/task_id=stg_intl_user_useroptins.get_count/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2026-03-10 02:44:02.200313 [warning ] Failed to send heartbeat. Will be retried [supervisor] failed_heartbeats=1 max_retries=3 ti_id=UUID('019cd54c-28b0-7e18-9a7b-71ba469bf545')
[31mΓò¡ΓöÇ[0m[31mΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ[0m[31m [0m[1;31mTraceback [0m[1;2;31m(most recent call last)[0m[31m [0m[31mΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ[0m[31mΓöÇΓò«[0m
[31mΓöé[0m [2;33m/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/[0m[1;33mdefault.py[0m:[94m101[0m in [31mΓöé[0m
[31mΓöé[0m [92mmap_httpcore_exceptions[0m
this is the worker logs which we see the error exact at 02:44 am
API server logs @02:45 am UTC
2026-03-10 02:45:23 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=155023 ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
[2026-03-10T02:45:23.575+0000] {exceptions.py:77} ERROR - Error with id 9zBmdizJ
and the error that I have shared in the earlier thread and also I see below error
2026-03-10 02:45:23 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=155023 ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
[2026-03-10T02:45:23.575+0000] {exceptions.py:77} ERROR - Error with id 9zBmdizJ
Airflow logs show below only there is no error
[2026-03-08, 08:16:29] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:30] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:31] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:32] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:33] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:34] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:35] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:36] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:37] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:38] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:39] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:40] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:41] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:42] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:43] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:44] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:45] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager"
[2026-03-08, 08:16:46] INFO - Waiting for container 'spark-kubernetes-driver' state to be completed: source="airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager
and after that task got failed with no error recorded in the logs
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16092145
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
GitHub user boring-cyborg[bot] added a comment to the discussion: Airflow task failed but spark kube app is running
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16073871
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Okay...only the point is reducing the concurrency will not cause any delay in the day to day task or scheduled job execution time not get increase ?
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16094580
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow task failed but spark kube app is running
Hopefully someone will
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16074835
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
@rcrchawla Thanks for the details. Reducing schedulers from 2 to 1 may reduce some DB activity, it won’t fix your issue since it seems to be heartbeat/API related
can you try reducing worker_concurrency from 16 to 8 or 10 (unfortunately behavior in Airflow 3.0.4 is not always comparable to 2.2.2 at the same concurrency)
you can also try increasing task_instance_heartbeat_timeout from 900 to 1800
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16092752
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Sure. I have suggested the above point to deployment team, they are currently considering to the point and will keep you posted about the further updates.
Also today I have noticed in our api server same mysql issue occur @02:47 am UTC which occured yesterday, in worker logs I see the below logs of it
2026-03-13 02:45:42.277543 [info ] [2ae514fc-eb08-4617-86b9-ae92e5a95011] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('019ce4a7-62a5-7a9d-9004-06b3f1f73368'), task_id='etl_load_to_ods_group_id_2.reconcile_meta_group_id_2', dag_id='segment-groupid-2', run_id='scheduled__2026-03-13T00:45:00+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=9, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('segment_groupid_2.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=segment-groupid-2/run_id=scheduled__2026-03-13T00:45:00+00:00/task_id=etl_load_to_ods_group_id_2.reconcile_meta_group_id_2/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2026-03-13 02:45:42.309091 [info ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
Till 02:45 **2ae514fc-eb08-4617-86b9-ae92e5a95011** runs fine
2026-03-13 02:46:00.160867 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:00.342989 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:00.575250 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:02.209653 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:02.780170 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:03.383007 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:03.855494 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:05.441315 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:05.606781 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:06.091219 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:07.350774 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:07.940790 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:09.082668 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:09.262903 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:11.170876 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:11.614824 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:12.320942 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:13.038009 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:13.884969 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:15.234633 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:15.896211 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client]
2026-03-13 02:46:17.023386 [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
[31mΓò░ΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓò»[0m
[1;91mReadTimeout: [0mtimed out
**2026-03-13 02:47:00.136773 [error ] Task execute_workload[2ae514fc-eb08-4617-86b9-ae92e5a95011] raised unexpected: ReadTimeout('timed out') [celery.app.trace]**
[31mΓò¡ΓöÇ[0m[31mΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇΓöÇ[0m[31m [0m[1;31mTraceback [0m[1;2;31m(most recent call last)[0m[31m [0m
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16111675
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Sure. It will be really helpful for us to get the solution for the issue mentioned above as everyday our production pipeline get failed due to it.
Thanks
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16073951
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow task failed but spark kube app is running
Moce to a discussion. You are not maintainer to open meta tasks, so please don't. Discussions are supposed to be used for such questions
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16073881
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
@rcrchawla
Try reducing pressure by decreasing the numbers of parallel tasks [celery] worker_concurrency
You can also try increasing task_instance_heartbeat_timeout
If the issue still persists, let me know if we can reproduce this in a minimal environment I'd be happy to take a look
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16092480
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
@shaealh
Task will not run slower by reducing the concurrency but only difference it will cause is increase in waiting time.
We can try with the above suggestion.
Thanks
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16096420
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Hi @shaealh
1. Full Mysql error text
2026-03-10 02:45:23 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=155023 ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
[2026-03-10T02:45:23.575+0000] {exceptions.py:77} ERROR - Error with id 9zBmdizJ
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 75, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 302, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 213, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cadwyn/structure/versions.py", line 474, in decorator
response = await self._convert_endpoint_response_to_version(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cadwyn/structure/versions.py", line 520, in _convert_endpoint_response_to_version
response_or_response_body: Union[FastapiResponse, object] = await run_in_threadpool(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
return await anyio.to_thread.run_sync(func)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2476, in run_sync_in_worker_thread
return await future
^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 967, in run
result = context.run(func, *args)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cadwyn/schema_generation.py", line 515, in __call__
return self._original_callable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/execution_api/routes/xcoms.py", line 419, in set_xcom
session.flush()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3449, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3588, in _flush
with util.safe_reraise():
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 3549, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
util.preloaded.orm_persistence.save_obj(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
_emit_insert_statements(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
c = connection._execute_20(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
util.raise_(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 179, in execute
res = self._query(mogrified_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
db.query(q)
File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/connections.py", line 280, in query
_mysql.connection.query(self, query)
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd518-d7c9-7e7e-bde2-efc6322e36a3
2026-03-10 02:45:23 [debug ] Heartbeat updated state=running ti_id=019cd54c-28ad-7db0-b0f8-d64ed0916d78
2026-03-10 02:45:23 [debug ] Retrieved current task state
2. Exact Airflow failure is not sure but from running it transition to failed and this for all the task either it is a bashoperator, speciesparkkubernetesoperator etc. And the specific time period only we see this issue around 2:30 am - 3:45 am UTC
3. Spark Kube operator task config
load_track = SparkKubernetesOperator(random_name_suffix=False,get_logs=False,delete_on_termination=False,
task_id=f'load_track_{group_value}',
namespace="dataengineering",
kubernetes_conn_id = 'kubernetes_default',
application_file=open(f"{ODS_LOAD_PATH}/load_to_snowflake/yamls/segment_sf_load_{group_value}.yaml").read(),
#application_file=open(f"{ETL_PATH}/segment_sf_load_{group_value}.yaml").read(),
do_xcom_push=False,
startup_timeout_seconds=2400,
on_failure_callback=custom_failure_function,
trigger_rule='none_failed_min_one_success',
dag=dag
)
**this is xcom related task**
load_track = SparkKubernetesOperator(random_name_suffix=False,get_logs=False,delete_on_termination=False,
task_id=f'load_track_{group_value}',
namespace="dataengineering",
kubernetes_conn_id = 'kubernetes_default',
application_file=open(f"{ODS_LOAD_PATH}/load_to_snowflake/yamls/segment_sf_load_{group_value}.yaml").read(),
#application_file=open(f"{ETL_PATH}/segment_sf_load_{group_value}.yaml").read(),
do_xcom_push=False,
startup_timeout_seconds=2400,
on_failure_callback=custom_failure_function,
trigger_rule='none_failed_min_one_success',
dag=dag
)
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16091804
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Okay currently we have de-scaled our scheduler from 2 to 1 will it help getting some performance improve. We do not have that kind of minimal environment where there is huge traffic of jobs.
Currently task_instance_heartbeat_timeout -- 900
and worker concurrency is 16 with 2 workers running earlier also we used to have 16 concurrency in airflow version 2.2.2 and we have upgraded to airflow 3.0.4.
Will de-scaling of scheduler server help ?
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16092588
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "rcrchawla (via GitHub)" <gi...@apache.org>.
GitHub user rcrchawla added a comment to the discussion: Airflow task failed but spark kube app is running
Hi @shaealh
As discussed we have lowered down the concurrency and it is working fine but we faced one challenge which is one time only where scheduler is launching a task (Spark Kubernetes Operator) which is working fine for few minutes and after that worker is unable to send the signal and it retried 5 times and worker got the message "Server error task instance is already running somewhere else" and where it launched another pid for same task _id now we see in the airflow logs this task wait for getting the resource to launch the driver where we have set the property "startup_timeout_seconds" to 40 minutes but in the logs after it waiting it killed the task now not sure why it has killed the task.
Could you please suggest anything by seeing the below summarized logs :
[Uploading 20260326.zip…]()
airflow-worker-1
2026-03-26 02:14 -- 173a1136-b645-4d67-a13a-a485a3704efc -- load_track_group_id_nerd_portals -- 019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:26.905924 [warning ] Failed to send heartbeat. Will be retried [supervisor] failed_heartbeats=1 max_retries=3 ti_id=UUID('019d277e-99e9-763d-b75b-d6cd75120881')
2026-03-26 02:47:57.527466 [warning ] Server error [airflow.sdk.api.client] detail={'detail': {'reason': 'not_running', 'message': 'TI is no longer in the running state and task should terminate', 'current_state': None}}
2026-03-26 02:47:57.527755 [error ] Server indicated the task shouldn't be running anymore [supervisor] detail={'detail': {'reason': 'not_running', 'message': 'TI is no longer in the running state and task should terminate', 'current_state': None}} status_code=409 ti_id=UUID('019d277e-99e9-763d-b75b-d6cd75120881')
2026-03-26 02:47:58.194844 [info ] Task execute_workload[3fb3128a-86c9-43bc-a1fb-3b0a6dcbcec4] received [celery.worker.strategy]
2026-03-26 02:47:58.228603 [info ] [3fb3128a-86c9-43bc-a1fb-3b0a6dcbcec4] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('019d277e-99e9-763d-b75b-d6cd75120881'), task_id='ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals', dag_id='segment-groupid-1', run_id='scheduled__2026-03-26T00:15:00+00:00', try_number=2, map_index=-1, pool_slots=1, queue='default', priority_weight=13, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=datetime.datetime(2026, 3, 26, 2, 14, 17, 330486, tzinfo=TzInfo(UTC))) dag_rel_path=PurePosixPath('segment_groupid_1.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=segment-groupid-1/run_id=scheduled__2026-03-26T00:15:00+00:00/task_id=ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals/attempt=2.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2026-03-26 02:47:58.256886 [info ] Secrets backends loaded for worker [supervisor]
2026-03-26 02:48:02.577882 [info ] Task execute_workload[173a1136-b645-4d67-a13a-a485a3704efc] succeeded in 2025.1995898000896s: None [celery.app.trace]
2026-03-26 02:48:10.794292 [info ] Task finished [supervisor] duration=12.537848281674087 exit_code=0 final_state=failed
2026-03-26 02:48:10.810689 [info ] Task execute_workload[3fb3128a-86c9-43bc-a1fb-3b0a6dcbcec4] succeeded in 12.614215551875532s: None [celery.app.trace]
2026-03-26 02:49:55.420123 [info ] Task finished [supervisor] duration=36.28673866484314 exit_code=0 final_state=success
--------------------------------------------------------------------------------------------------------
api_server
---------------------------------------------
2026-03-26 02:45:47 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1193430 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:45:47 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=1193430 state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:45:47 [debug ] Heartbeat updated state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:60936 - "PUT /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/heartbeat HTTP/1.1" 204 No Content
2026-03-26 02:46:38 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=1193430 state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:46:38 [debug ] Heartbeat updated state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:50958 - "PUT /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/heartbeat HTTP/1.1" 204 No Content
2026-03-26 02:46:58 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1193430 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:13 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1193430 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:21 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1193430 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:27 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1193430 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:55 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=1193430 state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:55 [debug ] Heartbeat updated state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
Taskk failed with below error
2026-03-26 02:47:57 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1193430 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:57 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=1193430 state=None ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:57 [warning ] Task not in running state current_state=None ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:54322 - "PUT /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/heartbeat HTTP/1.1" 409 Conflict
New task launched for same task id
2026-03-26 02:47:58 [debug ] Starting task instance run hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1201129 ti_id=019d277e-99e9-763d-b75b-d6cd75120881 unixname=airflow
2026-03-26 02:47:58 [debug ] Starting task instance run hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1201129 ti_id=019d277e-99e9-763d-b75b-d6cd75120881 unixname=airflow
2026-03-26 02:47:58 [debug ] Retrieved task instance details dag_id=segment-groupid-1 state=queued task_id=ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:58 [info ] Task started hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local previous_state=queued ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:58 [info ] Task instance state updated rows_affected=1 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:54334 - "PATCH /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/run HTTP/1.1" 200 OK
2026-03-26 02:47:58 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1201129 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:58 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=1201129 state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:47:58 [debug ] Heartbeat updated state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:54334 - "PUT /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/heartbeat HTTP/1.1" 204 No Content
2026-03-26 02:48:09 [debug ] Processing heartbeat hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local pid=1201129 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:48:09 [debug ] Retrieved current task state current_hostname=airflow-worker-1.airflow-worker.de-services.svc.cluster.local current_pid=1201129 state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:48:09 [debug ] Heartbeat updated state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:35238 - "PUT /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/heartbeat HTTP/1.1" 204 No Content
INFO: 172.26.145.209:35238 - "GET /execution/connections/smtp_default HTTP/1.1" 200 OK
2026-03-26 02:48:10 [debug ] Updating task instance state new_state=failed ti_id=019d277e-99e9-763d-b75b-d6cd75120881
2026-03-26 02:48:10 [debug ] Retrieved current task instance state max_tries=0 previous_state=running ti_id=019d277e-99e9-763d-b75b-d6cd75120881 try_number=2
2026-03-26 02:48:10 [debug ] Processing heartbeat hostname=airflow-worker-2.airflow-worker.de-services.svc.cluster.local pid=1128588 ti_id=019d278c-5124-76a1-895b-d904aa955a06
2026-03-26 02:48:10 [info ] Task instance state updated new_state=failed rows_affected=1 ti_id=019d277e-99e9-763d-b75b-d6cd75120881
INFO: 172.26.145.209:35238 - "PATCH /execution/task-instances/019d277e-99e9-763d-b75b-d6cd75120881/state HTTP/1.1" 204 No Content
scheduler
-------------
[2026-03-26T02:14:17.329+0000] {scheduler_job_runner.py:451} INFO - 2 tasks up for execution:
<TaskInstance: segment-groupid-2.etl_load_to_ods_group_id_2.branching_etl_step schedsuled__2026-03-26T00:45:00+00:00 [scheduled]>
<TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [scheduled]>
[2026-03-26T02:14:17.330+0000] {scheduler_job_runner.py:662} INFO - Setting the following tasks to queued state:
<TaskInstance: segment-groupid-2.etl_load_to_ods_group_id_2.branching_etl_step scheduled__2026-03-26T00:45:00+00:00 [scheduled]>
<TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [scheduled]>
[2026-03-26T02:14:17.333+0000] {scheduler_job_runner.py:768} INFO - Trying to enqueue tasks: [<TaskInstance: segment-groupid-2.etl_load_to_ods_group_id_2.branching_etl_step scheduled__2026-03-26T00:45:00+00:00 [scheduled]>, <TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [scheduled]>] for executor: CeleryExecutor(parallelism=32)
[2026-03-26T02:14:17.462+0000] {scheduler_job_runner.py:811} INFO - Received executor event with state queued for task instance TaskInstanceKey(dag_id='segment-groupid-1', task_id='ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals', run_id='scheduled__2026-03-26T00:15:00+00:00', try_number=1, map_index=-1)
[2026-03-26T02:47:55.913+0000] {scheduler_job_runner.py:2210} INFO - Reset the following 2 orphaned TaskInstances:
<TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [running]>
<TaskInstance: Document_pipeline_kube.CORE.document_parser scheduled__2026-03-26T00:30:00+00:00 [running]>
[2026-03-26T02:47:58.140+0000] {scheduler_job_runner.py:451} INFO - 2 tasks up for execution:
<TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [scheduled]>
[2026-03-26T02:47:58.141+0000] {scheduler_job_runner.py:662} INFO - Setting the following tasks to queued state:
<TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [scheduled]>
[2026-03-26T02:47:58.147+0000] {scheduler_job_runner.py:768} INFO - Trying to enqueue tasks: [<TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [scheduled]>, <TaskInstance: Document_pipeline_kube.CORE.document_parser scheduled__2026-03-26T00:30:00+00:00 [scheduled]>] for executor: CeleryExecutor(parallelism=32)
[2026-03-26T02:47:58.322+0000] {scheduler_job_runner.py:811} INFO - Received executor event with state queued for task instance TaskInstanceKey(dag_id='segment-groupid-1', task_id='ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals', run_id='scheduled__2026-03-26T00:15:00+00:00', try_number=2, map_index=-1)
[2026-03-26T02:47:58.336+0000] {scheduler_job_runner.py:843} INFO - Setting external_id for <TaskInstance: segment-groupid-1.ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals scheduled__2026-03-26T00:15:00+00:00 [running]> to 3fb3128a-86c9-43bc-a1fb-3b0a6dcbcec4
[2026-03-26T02:48:04.417+0000] {scheduler_job_runner.py:811} INFO - Received executor event with state success for task instance TaskInstanceKey(dag_id='segment-groupid-1', task_id='ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals', run_id='scheduled__2026-03-26T00:15:00+00:00', try_number=1, map_index=-1)
[2026-03-26T02:48:04.429+0000] {scheduler_job_runner.py:853} INFO - TaskInstance Finished: dag_id=segment-groupid-1, task_id=ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals, run_id=scheduled__2026-03-26T00:15:00+00:00, map_index=-1, run_start_date=2026-03-26 02:47:58.263013+00:00, run_end_date=None, run_duration=None, state=running, executor=CeleryExecutor(parallelism=32), executor_state=success, try_number=1, max_tries=0, pool=default_pool, queue=default, priority_weight=13, operator=SparkKubernetesOperator, queued_dttm=2026-03-26 02:47:58.142052+00:00, scheduled_dttm=2026-03-26 02:47:58.060945+00:00,queued_by_job_id=2414479, pid=1201129
[2026-03-26T02:48:12.148+0000] {scheduler_job_runner.py:811} INFO - Received executor event with state success for task instance TaskInstanceKey(dag_id='segment-groupid-1', task_id='ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals', run_id='scheduled__2026-03-26T00:15:00+00:00', try_number=2, map_index=-1)
[2026-03-26T02:48:12.159+0000] {scheduler_job_runner.py:853} INFO - TaskInstance Finished: dag_id=segment-groupid-1, task_id=ODS_Load_group_id_nerd_portals.load_track_group_id_nerd_portals, run_id=scheduled__2026-03-26T00:15:00+00:00, map_index=-1, run_start_date=2026-03-26 02:47:58.263013+00:00, run_end_date=2026-03-26 02:48:10.755668+00:00, run_duration=12.4927, state=failed, executor=CeleryExecutor(parallelism=32), executor_state=success, try_number=2, max_tries=0, pool=default_pool, queue=default, priority_weight=13, operator=SparkKubernetesOperator, queued_dttm=2026-03-26 02:47:58.142052+00:00, scheduled_dttm=2026-03-26 02:47:58.060945+00:00,queued_by_job_id=2414479, pid=1201129
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16461428
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
@rcrchawla
Yup although tasks don't run slower, reducing concurrency can increase waiting time before tasks start if the queue is full.
Your current capacity is 32 parallel slots (2 workers x 16)
If we move to 10, capacity becomes 20 slots.
Given current failures, I'd do that to test how it behaves with fewer heartbeat/ API failures, then tune throughput.
For example, I might try a lower number first like 8 and monitor if that fixes the issue. If it does then we've confirmed the root cause. Next I'd monitor jobs and run durations for a couple of days. If the delays are high that's when I'd play with the parameters to see what's ideal for this specific use case (eg. adding a working and gradually creasing the concurrency)
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16095553
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
Thank you @rcrchawla
This doesn't look like a Spark specific issue since you see the same failure with BashOperator, the problem is likely in Airflow control plane path (Execution API <--> metadata DB) not Spark runtime.
One more thing to confirm: I still don't see final DB error message
can you share the exact exception line after the stack trace, for example sqlalchemy.exc.OperationalError ... the final line in MySQL/SQLAlchemy error should be enough no need for the full traceback
Second thing that would help is the exact Airflow line that says why the task state changed from running to failed
So far this still looks more like DB /API capacity or transient DB availability
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16091889
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "shaealh (via GitHub)" <gi...@apache.org>.
GitHub user shaealh added a comment to the discussion: Airflow task failed but spark kube app is running
@rcrchawla Thanks for sharing the logs. This looks more like an Airflow control plane issue than a Spark app issue.
Here's why I say that:
• Repeated "Client.request" retries
• "Failed to send heartbeat"
• API server error in "set_xcom -> session.flush()" to MySQL
So Spark can keep running, but Airflow may still mark the task failed.
Could you share 3 things to confirm?
1) Full MySQL error text (after _mysql.connection.query(...))
2) Exact Airflow task failure reason or state transition
3) Spark operator /XCom-related task config
If this reproduces with healthy DB/ network, then it likely needs an Airflow fix. Otherwise it’s probably infra/DB pressure
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16080166
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow task failed but spark kube app is running [airflow]
Posted by "hezeclark (via GitHub)" <gi...@apache.org>.
GitHub user hezeclark added a comment to the discussion: Airflow task failed but spark kube app is running
This is a common issue with Airflow + SparkKubernetesOperator when the Airflow task timeout is shorter than the actual Spark job duration.
**Root cause**: Airflow marks the task as failed when it doesn't receive a heartbeat or when the task's `execution_timeout` is exceeded, but the Spark application keeps running in Kubernetes independently — it has no awareness of Airflow's state.
**Solutions:**
**1. Increase `execution_timeout` on the Spark task**
```python
from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator
from datetime import timedelta
submit_job = SparkKubernetesOperator(
task_id='submit_spark_job',
application='/path/to/spark-app.yaml',
execution_timeout=timedelta(hours=3), # longer than max expected job duration
...
)
```
**2. Use `SparkKubernetesSensor` to poll instead of waiting synchronously**
```python
from airflow.providers.cncf.kubernetes.sensors.spark_kubernetes import SparkKubernetesSensor
monitor_job = SparkKubernetesSensor(
task_id='monitor_spark_job',
application_name='{{ task_instance.xcom_pull(task_ids=\"submit_spark_job\") }}',
poke_interval=30,
timeout=7200, # 2 hours
...
)
```
**3. Add a cleanup DAG / task that deletes orphaned Spark apps**
When Airflow fails but Spark keeps running, you need to handle the orphaned app. Add an `on_failure_callback` that calls `kubectl delete sparkapplication <name>` to prevent resource leaks:
```python
def cleanup_spark_app(context):
import subprocess
app_name = context['task_instance'].xcom_pull(task_ids='submit_spark_job')
subprocess.run(['kubectl', 'delete', 'sparkapplication', app_name, '-n', 'spark'], check=False)
submit_job = SparkKubernetesOperator(
on_failure_callback=cleanup_spark_app,
...
)
```
GitHub link: https://github.com/apache/airflow/discussions/63298#discussioncomment-16117745
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org