[jira] [Updated] (AIRFLOW-1108) DAG.fileloc is wrong when DAG is created from a factory method

["Ivan Vergiliev (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ivan Vergiliev updated AIRFLOW-1108:
------------------------------------
    Description: 
When using a factory method that creates DAGs, the created DAG's `fileloc` is set to the file that contains the factory method instead of the file that declares the DAG. For example:

{code:title=dag_factory.py}
def create_dag(dag_id):
  dag = DAG(dag_id=dag_id, [...])
  [...]
  return dag
{code}

{code:title=dag_from_factory.py}
dag1 = create_dag('dag1')
{code}

In this case, dag1.fileloc will be `dag_factory.py` instead of the correct `dag_from_factory.py`. This causes the Code tab to display the factory code instead of the definition code. More importantly though, it also causes problems when refreshing the DAG in `DagBag.get_dag` - it tries to reload the file that contains the factory method, however this file doesn't contain any DAG definitions itself. This causes the webserver to think that the DAG does not actually exist.

AFAICT this is a regression from this fix for SubDAGs: https://issues.apache.org/jira/browse/AIRFLOW-365 . I'm attaching a patch that adds a test for this. (Not creating it as a PR as it will obviously cause the unit tests to fail.) If I have time, I might try fixing it as well.

  was:
When using a factory method that creates DAGs, the created DAG's `fileloc` is set to the file that contains the factory method instead of the file that declares the DAG. For example:

# dag_factory.py
    def create_dag(dag_id):
       dag = DAG(dag_id=dag_id, [...])
       [...]
       return dag

# dag_from_factory.py
    dag1 = create_dag('dag1')

In this case, dag1.fileloc will be `dag_factory.py` instead of the correct `dag_from_factory.py`. This causes the Code tab to display the factory code instead of the definition code. More importantly though, it also causes problems when refreshing the DAG in `DagBag.get_dag` - it tries to reload the file that contains the factory method, however this file doesn't contain any DAG definitions itself. This causes the webserver to think that the DAG does not actually exist.

AFAICT this is a regression from this fix for SubDAGs: https://issues.apache.org/jira/browse/AIRFLOW-365 . I'm attaching a patch that adds a test for this. (Not creating it as a PR as it will obviously cause the unit tests to fail.) If I have time, I might try fixing it as well.


> DAG.fileloc is wrong when DAG is created from a factory method
> --------------------------------------------------------------
>
>                 Key: AIRFLOW-1108
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1108
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: DAG, models
>    Affects Versions: Airflow 1.8
>            Reporter: Ivan Vergiliev
>         Attachments: dag_factory_fileloc.patch
>
>
> When using a factory method that creates DAGs, the created DAG's `fileloc` is set to the file that contains the factory method instead of the file that declares the DAG. For example:
> {code:title=dag_factory.py}
> def create_dag(dag_id):
>   dag = DAG(dag_id=dag_id, [...])
>   [...]
>   return dag
> {code}
> {code:title=dag_from_factory.py}
> dag1 = create_dag('dag1')
> {code}
> In this case, dag1.fileloc will be `dag_factory.py` instead of the correct `dag_from_factory.py`. This causes the Code tab to display the factory code instead of the definition code. More importantly though, it also causes problems when refreshing the DAG in `DagBag.get_dag` - it tries to reload the file that contains the factory method, however this file doesn't contain any DAG definitions itself. This causes the webserver to think that the DAG does not actually exist.
> AFAICT this is a regression from this fix for SubDAGs: https://issues.apache.org/jira/browse/AIRFLOW-365 . I'm attaching a patch that adds a test for this. (Not creating it as a PR as it will obviously cause the unit tests to fail.) If I have time, I might try fixing it as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)