[jira] [Updated] (AIRFLOW-2970) Kubernetes logging is broken

["Steven Miller (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steven Miller updated AIRFLOW-2970:
-----------------------------------
    Attachment: image-2019-09-17-12-19-45-207.png

> Kubernetes logging is broken
> ----------------------------
>
>                 Key: AIRFLOW-2970
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2970
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: executors
>            Reporter: Jon Davies
>            Assignee: Daniel Imberman
>            Priority: Major
>         Attachments: image-2019-09-17-12-19-45-207.png
>
>
> I'm using Airflow with the Kubernetes executor and pod operator. And my DAGs are configured to do get_log=True and all my DAGs are set to log to stdout and I can see all the logs in kubectl logs.
> I can see that the scheduler logs things to: $AIRFLOW_HOME/logs/scheduler/2018-08-28/*
> However, this just consists of:
> {code:java}
> [2018-08-28 13:03:27,695] {jobs.py:385} INFO - Started process (PID=16994) to work on /home/airflow/dags/dag.py
> [2018-08-28 13:03:27,697] {jobs.py:1782} INFO - Processing file /home/airflow/dags/dag.py for tasks to queue
> [2018-08-28 13:03:27,697] {logging_mixin.py:95} INFO - [2018-08-28 13:03:27,697] {models.py:258} INFO - Filling up the DagBag from /home/airflow/dags/dag.py
> {code}
> If I quickly exec into the executor the scheduler spins up, I can see that things are properly logged to:
> {code:java}
> /home/airflow/logs/dag$ tail -f dag-downloader/2018-08-28T13\:05\:07.704072+00\:00/1.log
> [2018-08-28 13:05:24,399] {logging_mixin.py:95} INFO - [2018-08-28 13:05:24,399] {pod_launcher.py:112} INFO - Event: dag-downloader-015ca48c had an event of type Pending
> ...
> [2018-08-28 13:05:37,193] {logging_mixin.py:95} INFO - [2018-08-28 13:05:37,193] {pod_launcher.py:95} INFO - b'INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (7): blah-blah.s3.eu-west-1.amazonaws.com\n'
> ...
> ...all other log lines from pod...
> {code}
> However, this executor pod only exists for the duration of the lifetime of the task pod so the logs are lost pretty much immediately after the task runs. There is nothing that ships the logs back to the scheduler and/or web UI.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)