[jira] [Updated] (AIRFLOW-4171) KubernetesPodOperator / pod_launcher cannot capture entire log after the task complete

["Kaxil Naik (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-4171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kaxil Naik updated AIRFLOW-4171:
--------------------------------
    Fix Version/s:     (was: 1.10.2)

> KubernetesPodOperator / pod_launcher cannot capture entire log after the task complete
> --------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-4171
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4171
>             Project: Apache Airflow
>          Issue Type: Bug
>    Affects Versions: 1.10.2
>            Reporter: Jiaxun Song
>            Priority: Major
>
> While using KubernetesPodOperator, I found that even though get_logs was set to be True, only a few lines of log from the task Pod is captured. Following is the potential explanation.
> In the following pod_launcher module class PodLauncher
> [https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
> function run_pod returns self._monitor_pod(pod, get_logs). And inside function _monitor_pod, it only captures the log when the pod starts to run. In other words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a task, it can only capture the log at the very beginning of the Pod. So we cannot see the entire log from the Pod when the Pod completed.
>  Also, every time I got this message from the task log: '{{failed to watch file "/var/log/pods/afaa3c1c-5126-11e9-9808-0a91764318bc/base/0.log": no space left on device}}'. Maybe the problem is from K8S host VM (I am using AKS).
> Please correct me if I am wrong, but I tried everything to capture the entire log from the task Pod. The only explanation I can find is the above.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)