You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Hamid Mahmood (Jira)" <ji...@apache.org> on 2020/01/13 14:39:00 UTC
[jira] [Updated] (AIRFLOW-6551) KubernetesExecutor does not create
dynamic pods for tasks inside subdag
[ https://issues.apache.org/jira/browse/AIRFLOW-6551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hamid Mahmood updated AIRFLOW-6551:
-----------------------------------
Description:
KubernetesExecutor does not create dynamic pods for tasks inside subdag.
I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators inside a main_dag. Following is the hierarchy
* main_dag:
** subdagA
*** taskA1
*** taskA2
*** taskA3
** subdagB
*** subdagB_1
**** taskB1
**** taskB2
** task_main1
** task_main2
** task_main3
I have set the following parameter
{code:java}
AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
{code}
When I run the workflow main_dag, only a single pod is created with name subdagA-
309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) run inside that single pod.
In theory they should be 3 separate pods for each task inside subdagA.
Same is the case for subdagB, only a single pod is created to run subdagB, subdagB_1 and its tasks runs inside that pod.
But the tasks (task_main1, task_main2, task_main3) that are not inside further subdag runs in dynamic pods.
*Following is the log*
was:
Hi Everyone,
KubernetesExecutor does not create dynamic pods for tasks inside subdag.
I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators inside a main_dag. Following is the hierarchy
* main_dag:
** subdagA
*** taskA1
*** taskA2
*** taskA3
** subdagB
*** subdagB_1
**** taskB1
**** taskB2
** task_main1
** task_main2
** task_main3
I have set the following parameter
{code:java}
AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
{code}
When I run the workflow main_dag, only a single pod is created with name subdagA-
309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) run inside that single pod.
In theory they should be 3 separate pods for each task inside subdagA.
Same is the case for subdagB, only a single pod is created to run subdagB, subdagB_1 and its tasks runs inside that pod.
But the tasks (task_main1, task_main2, task_main3) that are not inside further subdag runs in dynamic pods.
*Following is the log*
> KubernetesExecutor does not create dynamic pods for tasks inside subdag
> -----------------------------------------------------------------------
>
> Key: AIRFLOW-6551
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6551
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor-kubernetes
> Affects Versions: 1.10.7
> Reporter: Hamid Mahmood
> Assignee: Daniel Imberman
> Priority: Major
>
> KubernetesExecutor does not create dynamic pods for tasks inside subdag.
> I am running airflow 1.10.7 on eks 1.14. I have multiple subdags operators inside a main_dag. Following is the hierarchy
> * main_dag:
> ** subdagA
> *** taskA1
> *** taskA2
> *** taskA3
> ** subdagB
> *** subdagB_1
> **** taskB1
> **** taskB2
> ** task_main1
> ** task_main2
> ** task_main3
> I have set the following parameter
>
> {code:java}
> AIRFLOW__CORE__EXECUTOR = KubernetesExecutor
> {code}
> When I run the workflow main_dag, only a single pod is created with name subdagA-
> 309c4c564b9841529236a31dfaf135c5 and all the tasks (tasksA1,taskA2,taskA3) run inside that single pod.
> In theory they should be 3 separate pods for each task inside subdagA.
> Same is the case for subdagB, only a single pod is created to run subdagB, subdagB_1 and its tasks runs inside that pod.
> But the tasks (task_main1, task_main2, task_main3) that are not inside further subdag runs in dynamic pods.
>
> *Following is the log*
--
This message was sent by Atlassian Jira
(v8.3.4#803005)