[GitHub] [airflow] kobethuwis commented on issue #27772: Unable to get dag logs from Web UI

["kobethuwis (via GitHub)" <gi...@apache.org>]

kobethuwis commented on issue #27772:
URL: https://github.com/apache/airflow/issues/27772#issuecomment-1438390751

   Due to this issue, I tried understanding the way Airflow is logging tasks.
   
   > In the Airflow UI, remote logs take precedence over local logs when remote logging is enabled. If remote logs can not be found or accessed, local logs will be displayed. Note that logs are only sent to remote storage once a task is complete (including failure). In other words, remote logs for running tasks are unavailable (but local logs are available).
   
   The error log tells us that on the pod's level log retrieval is not possible, `/opt/airflow/logs/dag_id=demo_dag/run_id=manual__2022-11-17T09:46:26.538673+00:00/task_id=test-demo/attempt=1.log *** Fetching from: http://airflow-scheduler-599d84c9c9-r9kws:8793/log/dag_id=demo_dag/run_id=manual__2022-11-17T09:46:26.538673+00:00/task_id=test-demo/attempt=1.log `
   
   Which is due to the usage of the CeleryWorker and is elaborated in the documentation [here](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#serving-logs-from-workers) (... Airflow automatically starts an HTTP server to serve the logs ... The server is running on the port specified by worker_log_server_port option in [logging] section. By default, it is 8793 ...) This behaviour is confirmed by the second part of the error, where we are trying to fetch the from the server, which fails.
   
   I tried pinging/resolving the hostname inside the webserver, without result. This led me to believe that is a simply a DNS resolving error, which led me to this [issue
   ](https://stackoverflow.com/questions/62905221/dns-for-kubernetes-pods-and-airflow-worker-logs). 
   
   The suggested resolution works! This way I'm able to fall back to live local logs served by the worker for running tasks.
   
   ```
     - name: AIRFLOW__CORE__HOSTNAME_CALLABLE
       value: 'airflow.utils.net:get_host_ip_address'
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org