[jira] [Comment Edited] (AIRFLOW-4796) DOCO - DaskExecutor logs

["Darren Weber (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006987#comment-17006987 ] 

Darren Weber edited comment on AIRFLOW-4796 at 1/2/20 5:55 PM:
---------------------------------------------------------------

I don't know enough to propose any implementation details to solve this, I can only suggest the following tip to follow up on somehow.  If the airflow task has access to the dask-executor-client and possibly also a worker-id, then it might be able to use a client call to get the logs and parse them for relevant log details, e.g.

- [https://distributed.dask.org/en/latest/api.html#distributed.Client.get_worker_logs]

 

If Airflow is running in k8s and k8s is also running the dask cluster, it might be possible to define a k8s service endpoint, but that could just overcomplicate things without solving why the dask client does not retrieve the logs from the worker.  There might be something in Airflow that uses the s3-log config to override the log retrieval from the dask-worker?


was (Author: dazza):
I don't know enough to propose any implementation details to solve this, I can only suggest the following tip to follow up on somehow.  If the airflow task has access to the dask-executor-client and possibly also a worker-id, then it might be able to use a client call to get the logs and parse them for relevant log details, e.g.

- [https://distributed.dask.org/en/latest/api.html#distributed.Client.get_worker_logs]

> DOCO - DaskExecutor logs
> ------------------------
>
>                 Key: AIRFLOW-4796
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4796
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: executors, logging
>    Affects Versions: 1.10.3
>            Reporter: t oo
>            Priority: Major
>
> I have an Airflow installation (on Kubernetes). My setup uses {{DaskExecutor}}. I also configured remote logging to S3. However when the task is running I cannot see the log, and I get this error instead:
> *** Log file does not exist: /airflow/logs/dbt/run_dbt/2018-11-01T06:00:00+00:00/3.log
> *** Fetching from: http://airflow-worker-74d75ccd98-6g9h5:8793/log/dbt/run_dbt/2018-11-01T06:00:00+00:00/3.log
> *** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow-worker-74d75ccd98-6g9h5', port=8793): Max retries exceeded with url: /log/dbt/run_dbt/2018-11-01T06:00:00+00:00/3.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7d0668ae80>: Failed to establish a new connection: [Errno -2] Name or service not known',))
>  
> Once the task is done, the log is shown correctly.
> I believe what Airflow is doing is:
>  * for finished tasks read logs from s3
>  * for running tasks, connect to executor's _log server endpoint_ and show that.
> Looks like Airflow is using {{celery.worker_log_server_port}} to connect to my dask executor to fetch logs from there.
> h3. How to configure {{DaskExecutor}} to expose _log server endpoint_?
> my configuration:
>  
>  
> core remote_logging True 
> core remote_base_log_folder s3://some-s3-path
> core executor DaskExecutor 
> dask cluster_address 127.0.0.1:8786
> celery worker_log_server_port 8793 
>  
>  
> what i verified: - verified that the log file exists and is being written to on the executor while the task is running - called {{netstat -tunlp}} on executor container, but did not find any extra port exposed, where logs could be served from.
>  
>  
>  
> We solved the problem by simply starting a python HTTP handler on a worker.
> Dockerfile:
>  
> RUN mkdir -p $AIRFLOW_HOME/serve
> RUN ln -s $AIRFLOW_HOME/logs $AIRFLOW_HOME/serve/log
> worker.sh (run by Docker CMD):
>  
> #!/usr/bin/env bash
> cd $AIRFLOW_HOME/serve
> python3 -m http.server 8793 &
> cd -
> dask-worker $@
>  
>  
>  
> see [https://stackoverflow.com/questions/53121401/airflow-live-executor-logs-with-daskexecutor]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)