Re: Python DAG files must include an import from airflow for dag to be loaded to DagBag

[Georg Walther <ge...@markovian.com>]

Hi Tamara,


do you happen to use the celery executor for deploying your DAG factory?
We've got a setup where webserver, scheduler, and workers all run in one
Docker container each.
Our DAG factory is akin to this:

# factory.py
from airflow import DAG

def callable_():
    dag_id = ...
    parameters = (fetch parameters from a separate metadata database)
    dag = DAG(dag_id=dag_id)
    operators = (dag=dag, parameters=parameters...)

    globals[dag_id] = dag

The above callable_ is the Python callable of the PythonOperator of one
classically defined DAG in the `dags_folder` defined in our airflow.cfg.

We do have the required `from airport import DAG` statement in our DAG
factory, however the above `dag` object is not executed.


Thanks,

Georg


---
Dr. Georg Walther
Co-Founder of Markovian

Skype: georg.walther@markovian.com
WWW: https://markovian.com/


On Thu, Nov 17, 2016 at 11:45 AM, Tamara Mendt <tm...@hellofresh.com> wrote:

> Thanks very much for the detailed reply Maxime,
>
> Like you suggested, I had solved the issue by adding an unused import of an
> airflow class. It took me a while to get to this solution though. I had to
> look into the airflow code to the place you pointed to realize that this
> import was necessary to add the dag to the DagBag. Perhaps its worth it to
> add something about this in the Airflow documentation? Or perhaps adding an
> INFO log stating this sanity check, so that on running the airflow
> list_dags
> command, it is obvious that DAG files must contain airflow imports.
>
> Cheers,
>
> Tamara
>
> --
>
> [image: logo]
>   <http://www.facebook.com/hellofreshde>   <http://twitter.com/
> HelloFreshde>
>    <http://instagram.com/hellofreshde/>   <http://blog.hellofresh.de/>
> <https://app.adjust.com/ayje08?campaign=Hellofresh&
> deep_link=hellofresh%3A%2F%2F&post_deep_link=https%3A%2F%
> 2Fwww.hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_
> source%3Demail_signature&fallback=https%3A%2F%2Fwww.
> hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_source%
> 3Demail_signature>
>
> *HelloFresh App –Download Now!*
> <https://app.adjust.com/ayje08?campaign=Hellofresh&
> deep_link=hellofresh%3A%2F%2F&post_deep_link=https%3A%2F%
> 2Fwww.hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_
> source%3Demail_signature&fallback=https%3A%2F%2Fwww.
> hellofresh.com%2Fapp%2F%3Futm_medium%3Demail%26utm_source%
> 3Demail_signature>
> *We're active in:*
> US <https://www.hellofresh.com/?utm_medium=email&utm_source=
> email_signature>
>  |  DE
> <https://www.hellofresh.de/?utm_medium=email&utm_source=email_signature> |
> UK
> <https://www.hellofresh.co.uk/?utm_medium=email&utm_source=email_signature
> >
> |  NL
> <https://www.hellofresh.nl/?utm_medium=email&utm_source=email_signature> |
> AU
> <https://www.hellofresh.com.au/?utm_medium=email&utm_
> source=email_signature>
>  |  BE
> <https://www.hellofresh.be/?utm_medium=email&utm_source=email_signature> |
> AT <https://www.hellofresh.at/?utm_medium=email&utm_source=email_signature
> >
> |  CH
> <https://www.hellofresh.ch/?utm_medium=email&utm_source=email_signature> |
> CA <https://www.hellofresh.ca/?utm_medium=email&utm_source=email_signature
> >
>
> www.HelloFreshGroup.com
> <http://www.hellofreshgroup.com/?utm_medium=email&utm_
> source=email_signature>
>
> We are hiring around the world – Click here to join us
> <https://www.hellofresh.com/jobs/?utm_medium=email&utm_
> source=email_signature>
>
> --
>
> <https://www.hellofresh.com/jobs/?utm_medium=email&utm_
> source=email_signature>
> HelloFresh AG, Berlin (Sitz der Gesellschaft) | Vorstände: Dominik S.
> Richter (Vorsitzender), Thomas W. Griesel, Christian Gärtner | Vorsitzender
> des Aufsichtsrats: Jeffrey Lieberman | Eingetragen beim Amtsgericht
> Charlottenburg, HRB 171666 B | USt-Id Nr.: DE 302210417
>
> *CONFIDENTIALITY NOTICE:* This message (including any attachments) is
> confidential and may be privileged. It may be read, copied and used only by
> the intended recipient. If you have received it in error please contact the
> sender (by return e-mail) immediately and delete this message. Any
> unauthorized use or dissemination of this message in whole or in parts is
> strictly prohibited.
>