[D] How to disable component file logging (scheduler, worker) and log only to stdout/stderr? [airflow]

["bun-paul-kwon (via GitHub)" <gi...@apache.org>]

GitHub user bun-paul-kwon created a discussion: How to disable component file logging (scheduler, worker) and log only to stdout/stderr?

Hello Airflow community,

We are currently using AWS MWAA (Celery Executor) and considering migrating to Airflow 3.0.2 on Kubernetes (AWS EKS).

- Image: apache/airflow:3.0.2
- Helm chart: [https://airflow.apache.org airflow 1.18.0](https://airflow.apache.org/docs/helm-chart/stable/index.html)
- [values.yaml](https://github.com/apache/airflow/blob/helm-chart/1.18.0/chart/values.yaml)

Our **task logs** are configured to use remote logging to S3, and this is working well. Our configuration(`values.yaml`) for this is:
```yaml
executor: "CeleryExecutor"

logging:
  logging_level: "INFO"
  delete_local_logs: "True"
  remote_logging: "True"
  remote_base_log_folder: s3://<MY_AWS_S3_BUCKET>
  remote_log_conn_id: <MY_AIRFLOW_CONNECTION_NAME>

scheduler:
  logGroomerSidecar: false

dagProcessor: 
  logGroomerSidecar: false

workers:
  logGroomerSidecar: false

# ...
```

Our question is about the logs for **Airflow components (e.g., scheduler, dag-processor, worker)**, not the task logs.

We've observed the following behavior in the Helm chart:
1. The `apiserver` logs only to `stdout`/`stderr` (which is our desired behavior).
2. The `scheduler`, `dag-processor` and `worker` pods log to both `stdout`/`stderr` AND local files (in `/opt/airflow/logs`).
3. Because of this, the `logGroomerSidecar` is needed to clean up these local files.

Our goal is to **disable local file logging** for all components (scheduler, worker, dag-processor) and have them **log only to** `stdout`/`stderr`.

We use Datadog autodiscovery, which scrapes these `stdout`/`stderr` streams for logging. The local file logs are redundant for us, and we would like to disable the `logGroomerSidecar` container entirely.

Is there a recommended way to configure this using the Helm chart `values.yaml`?

Thank you for your guidance!

GitHub link: https://github.com/apache/airflow/discussions/57535

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] How to disable component file logging (scheduler, worker) and log only to stdout/stderr? [airflow]

["bun-paul-kwon (via GitHub)" <gi...@apache.org>]

GitHub user bun-paul-kwon added a comment to the discussion: How to disable component file logging (scheduler, worker) and log only to stdout/stderr?

Hello, @potiuk 

Excuse me, I was wondering if you could provide some help.

I'm trying to find out if it's possible to output all logs for `scheduler`, `worker`, and `dag-processor` components directly to stdout/stderr instead of writing to files.

If you have any information or guidance on this, I'd greatly appreciate it! 🙏

Thank you!

GitHub link: https://github.com/apache/airflow/discussions/57535#discussioncomment-14879159

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org