You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jiaxun Song (JIRA)" <ji...@apache.org> on 2019/03/28 06:05:01 UTC
[jira] [Updated] (AIRFLOW-4171) KubernetesPodOperator /
pod_launcher cannot capture entire log after the task complete
[ https://issues.apache.org/jira/browse/AIRFLOW-4171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiaxun Song updated AIRFLOW-4171:
---------------------------------
Description:
While using KubernetesPodOperator, I found that even though get_logs was set to be True, only a few lines of log from the task Pod is captured. Following is the potential explanation.
In the following pod_launcher module class PodLauncher
[https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
function run_pod returns self._monitor_pod(pod, get_logs). And inside function _monitor_pod, it only captures the log when the pod starts to run. In other words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a task, it can only capture the log at the very beginning of the Pod. So we cannot see the entire log from the Pod when the Pod completed.
Please correct me if I am wrong, but I tried everything to capture the entire log from the task Pod. The only explanation I can find is the above.
was:
While using KubernetesPodOperator, I found that even though get_logs was set to be True, only a few lines of log from the task Pod is captured. Following is the potential explanation.
In the following pod_launcher module
[https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
function run_pod returns self._monitor_pod(pod, get_logs). And inside function _monitor_pod, it only captures the log when the pod starts to run. In other words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a task, it can only capture the log at the very beginning of the Pod. So we cannot see the entire log from the Pod when the Pod completed.
Please correct me if I am wrong, but I tried everything to capture the entire log from the task Pod. The only explanation I can find is the above.
> KubernetesPodOperator / pod_launcher cannot capture entire log after the task complete
> --------------------------------------------------------------------------------------
>
> Key: AIRFLOW-4171
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4171
> Project: Apache Airflow
> Issue Type: Bug
> Affects Versions: 1.10.2
> Reporter: Jiaxun Song
> Priority: Major
> Fix For: 1.10.2
>
>
> While using KubernetesPodOperator, I found that even though get_logs was set to be True, only a few lines of log from the task Pod is captured. Following is the potential explanation.
> In the following pod_launcher module class PodLauncher
> [https://github.com/apache/airflow/blob/master/airflow/contrib/kubernetes/pod_launcher.py]
> function run_pod returns self._monitor_pod(pod, get_logs). And inside function _monitor_pod, it only captures the log when the pod starts to run. In other words, when KubernetesPodOperator is using pod_launcher to spin up a Pod for a task, it can only capture the log at the very beginning of the Pod. So we cannot see the entire log from the Pod when the Pod completed.
>
> Please correct me if I am wrong, but I tried everything to capture the entire log from the task Pod. The only explanation I can find is the above.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)