How to work collaboratively on the same DAGs in Airflow

[Andreas Monger <an...@gmail.com>]

Hi Everyone,

we are facing issues, when trying to smoothly work collaboratively on the
same DAGs in the same (development) environment on Airflow.

As of now, we have a common code-base for our project, where multiple users
in various teams are making changes at the same time.
This leads to problems and conflicts when regularly pushing changes to
airflow. So we would like to be able to work in parallel and in isolation.

I thought I was smart and tried to enable it by:
• Enabling relative imports
• Create a user-specific ansible deployment that creates a new DAG folder
per user and adds a dag-id prefix per user
While the ansible stuff works nicely, I didn’t manage to convince Airflow
to use relative imports.
Also failed with
https://airflow.apache.org/concepts.html#packaged-dags approach,
so far… but it might as well be due my limited understanding of how Airflow
imports DAGs.

The most promising folder structure so far looked like this:
{airflow-dag-root}/my_project/
                          + __init__.py
                          + my_dag.py
                          + utils/
                              + __init__.py
                              + util.py

When trying to import like this in my_dag.py:
   from .utils.util import my_helper
... I get this error: "attempted relative import with no known parent
package"

Do you have any suggestions on how to achieve relative imports or another
nice way of working in isolation in airflow?
Thanks!

Best, Andreas