You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Usuychik (via GitHub)" <gi...@apache.org> on 2025/12/18 13:20:01 UTC
[I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Usuychik opened a new issue, #59602:
URL: https://github.com/apache/airflow/issues/59602
### Apache Airflow version
3.1.5
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
We have Airflow 3.1.5 deployed in kubernetes using official helm. Configured remote logging and celery workers autoscaling:
`workers:
# Number of airflow celery workers in StatefulSet
replicas: 1
resources:
requests:
cpu: 1
memory: 9Gi
persistence:
# Enable persistent volumes
enabled: true
# Volume size for worker StatefulSet
size: 50Gi
keda:
enabled: true
cooldownPeriod: 600
minReplicaCount: 1
maxReplicaCount: 10`
`config:
logging:
remote_logging: True
delete_local_logs: True
remote_log_conn_id: airflow-logs
remote_base_log_folder: s3://yyyyyy/xxxxx`
And when we try yo check logs for failed dags it shows error like this:
`Log message source details sources=["Could not read served logs: HTTPConnectionPool(host='airflow-worker-2.airflow-worker.xxxx.svc.cluster.local', port=8793): Max retries exceeded with url: /log/dag_id=xxxx/run_id=manual__xxx/task_id=yyy/map_index=25/attempt=8.log (Caused by NameResolutionError(\"HTTPConnection(host='airflow-worker-2.airflow-worker.xxxx.svc.cluster.local', port=8793): Failed to resolve 'airflow-worker-2.airflow-worker.airflow-ai.svc.cluster.local' ([Errno -2] Name or service not known)\"))"]`
In general there are no such service at all. There are only airflow-worker.xxxx.svc.cluster.local
### What you think should happen instead?
_No response_
### How to reproduce
Airflow 3.1.5 deployed in kubernetes using official helm. Configured remote logging and celery workers autoscaling. Try to run multiple dags so keda will scale up and than scale down. DAGs should fail somehow at that time.
### Operating System
gcloud k8s ubuntu image
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #59602: On failed DAGs logs are not displayed with remote logging and and celery workers scaling.
URL: https://github.com/apache/airflow/issues/59602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3755667191
@Usuychik Is the issue only to do with logs for in-progress tasks, or are logs for completed tasks not showing from S3 either?
Do you see any log files created in your S3 bucket for completed tasks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "sania-2912 (via GitHub)" <gi...@apache.org>.
sania-2912 commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3670329232
If the issue is open I would like to work on it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3763321650
Even when using CeleryExecutor with KubernetesPodOperator, it's still CeleryExecutor's responsibility to upload the logs. For each task (no matter what kind of operator you are using, even with KubernetesPodOperator), we create a TaskSDK subprocess via `supervise` call. Whether the logs are uploaded successfully or not depends on the TaskSDK subprocess.
https://github.com/apache/airflow/blob/7524107bf5ca6254abf43e024c33c1fb5e1dd7c6/providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py#L162-L170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "dispensable (via GitHub)" <gi...@apache.org>.
dispensable commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3803728780
The airflow api server requests log from airflow worker with worker podname (which can not be solved in airflow-worker deployment mode). so we should change the worker deploy to statefulset.
but statefuleset will mount a pvc for logs persistent which is useless when s3 remote logging enabled.
so all we need are:
1. enable statefulset for worker deployment
2. disable PVC generated by persistent value config
3. enable loggroomer sidecar to clean logs periodic
by use helm `--post-renderer` script with kustomize and helmfiles i find a working around:
for 2
```yaml
jsonPatches:
- target:
group: apps
version: v1
kind: StatefulSet
name: <name>-worker
patch:
- op: remove
path: /spec/volumeClaimTemplates
```
and in chart values.yaml
for 1 and 3
```yaml
workers:
persistence:
# Enable persistent volumes
enabled: true
logGroomerSidecar:
enabled: true
retentionDays: 1
frequencyMinutes: 60
extraVolumes:
- name: logs
emptyDir: {}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3841787089
If you have a solution for that , PR would be great @dispensable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3955546423
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "cdevacc1 (via GitHub)" <gi...@apache.org>.
cdevacc1 commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3690999533
I have the same issue with remote logging to gcs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "Usuychik (via GitHub)" <gi...@apache.org>.
Usuychik commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3759542482
@ashb - Checked, on S3 this one failed logs non exists. Is it possible that worker pod removed before S3 upload? In general I think in case of remote logging configured it should make or partial uploads, or try to upload all logs before shutdown (preshutdown hook or something like this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3670268425
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "sania-2912 (via GitHub)" <gi...@apache.org>.
sania-2912 commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3670327706
From my understanding, in Airflow 3.1.5 the webserver attempts to fetch served logs from a specific worker pod hostname (for example `airflow-worker-2...`), which may no longer exist after KEDA scales workers down. Even with `remote_logging=True` and `delete_local_logs=True`, the UI still appears to rely on the worker’s hostname, leading to DNS resolution failures. This behavior did not occur with the same setup on Airflow 2.10, which suggests a possible regression or behavioral change in Airflow 3.x logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "Usuychik (via GitHub)" <gi...@apache.org>.
Usuychik commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3759828027
> > or try to upload all logs before shutdown
>
> This would be ideal case. However, it’s possible the pod was terminated before the worker process finished uploading the log.
>
> [airflow/task-sdk/src/airflow/sdk/execution_time/supervisor.py](https://github.com/apache/airflow/blob/7bb3f203c24fca914cd296af47d6f00fa735feba/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1006-L1010)
>
> Lines 1006 to 1010 in [7bb3f20](/apache/airflow/commit/7bb3f203c24fca914cd296af47d6f00fa735feba)
> # Now at the last possible moment, when all logs and comms with the subprocess has finished, lets
> # upload the remote logs
> self._upload_logs()
>
> return self._exit_code
But its about task itself. And I am speaking about CelertyExecutor shutdown feature.
In general as for me Airflow miss some lightweight Celery Worker for k8s -- which will store all task task info in Redis so it will be able to get info on restart or to be captured by other executor replicas, in case if existing one executor removed. Airflow K8S executor in general require image to be based on Airflow image. For our use case we are using CeleryWorker executor to run DAGs which creates custom pods with something like:
`
kubernetes_task = KubernetesPodOperator(
task_id=k8s_task_id,
image=IMAGE,
cmds=["python3", "-u", "xxxx.py"],
arguments=args,
namespace="airflow",
in_cluster=True,
image_pull_secrets=[k8s.V1LocalObjectReference("xxxx")],
env_vars=base_env,
configmaps=["xxxx"],
container_resources=k8s.V1ResourceRequirements(
requests={"memory": "1Gi", "cpu": "0.5"}
),
image_pull_policy="Always",
deferrable=False,
get_logs=True,
dag=mydag,
logging_interval=5,
)
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3917780394
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3671360696
feel free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "Usuychik (via GitHub)" <gi...@apache.org>.
Usuychik commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3755829431
On successful dags it always ok. But for failed one in some cases it gives error like this. Will try to check next time if logs inside bucket exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] On failed DAGs logs are not displayed with remote logging and and celery workers scaling. [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on issue #59602:
URL: https://github.com/apache/airflow/issues/59602#issuecomment-3759651180
> or try to upload all logs before shutdown
This would be ideal case. However, it’s possible the pod was terminated before the worker process finished uploading the log.
https://github.com/apache/airflow/blob/7bb3f203c24fca914cd296af47d6f00fa735feba/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L1006-L1010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org