You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Michal Malus (JIRA)" <ji...@apache.org> on 2018/08/17 14:56:00 UTC
[jira] [Updated] (AIRFLOW-2914) Remote Workers execute DAG planning part

     [ https://issues.apache.org/jira/browse/AIRFLOW-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michal Malus updated AIRFLOW-2914:
----------------------------------
    Description: 
Hi,

 

During the execution phase remote Workers load (execute) the DAG definition file. Can this behavior be changed (configurable)?

This does not work when the scheduler node plans execution dynamically based on some configuration files not available to workers.

If the worker node is not able to recreate the same graph (and to do so access to configuration files is required, that only the scheduler node has) the execution fails.

 

More details with an example code:

[https://stackoverflow.com/questions/51867857/dynamic-airflow-dags-created-by-an-operator]

 

Best Regards,

Michal Malus

  was:
Hi,

 

During the execution phase remote Workers load (execute) the DAG definition file. Can this behavior be changed (configurable)?

This does not work when the scheduler node plans execution dynamically based on some configuration files not available by the workers.

If the worker node is not able to recreate the same graph (and to do so access to configuration files is required, that only the scheduler node has) the execution fails.

 

More details with an example code:

[https://stackoverflow.com/questions/51867857/dynamic-airflow-dags-created-by-an-operator]

 

Best Regards,

Michal Malus


> Remote Workers execute DAG planning part
> ----------------------------------------
>
>                 Key: AIRFLOW-2914
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2914
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: Dataflow, worker
>    Affects Versions: Airflow 1.9.0
>            Reporter: Michal Malus
>            Priority: Major
>
> Hi,
>  
> During the execution phase remote Workers load (execute) the DAG definition file. Can this behavior be changed (configurable)?
> This does not work when the scheduler node plans execution dynamically based on some configuration files not available to workers.
> If the worker node is not able to recreate the same graph (and to do so access to configuration files is required, that only the scheduler node has) the execution fails.
>  
> More details with an example code:
> [https://stackoverflow.com/questions/51867857/dynamic-airflow-dags-created-by-an-operator]
>  
> Best Regards,
> Michal Malus



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)