[jira] [Commented] (AIRFLOW-2371) SubdagOperator ignores pooling

["Matthew Bowden (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487558#comment-16487558 ] 

Matthew Bowden commented on AIRFLOW-2371:
-----------------------------------------

Just wanted to chime in and say that this issue is adversely affecting our workflow, and will require a workaround. We've leveraged subdags as a means of encapsulating work which can either be executed independently (as a regular DAG) or part of a larger dag (as a subdag). I think our workaround might be to have two separate pools, one for the SubDagOperator and one for the heavy processes within the subdag. That way, if the DAG is run alone, it will pull from the pool, whereas if it's run as a subdag it will still be limited by a pool. This prevents the subdag/subdag-task pool conflict issue, but unfortunately still means that the workers could be overrun by subdags and regular dag runs (since they don't share the same pool). 

> SubdagOperator ignores pooling
> ------------------------------
>
>                 Key: AIRFLOW-2371
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2371
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: subdag
>    Affects Versions: Airflow 1.9.0
>         Environment: Airflow 1.9.0
>            Reporter: Xiao Zhu
>            Priority: Major
>
> Basically we need a long term solution for https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor is a workaround, however it slows down execution of subdags.
> Looking for a better solution.
> I also see that when running subdags, db connection pools are not shared either (since they are separate processes), which causes db disconnection issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db because there can be too many connections. I hope it can be fixed too when subdag operator works with pooling..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)