Filtering DAGs in process_file

[Thomas Weise <th...@apache.org>]

Hi,

I want to include custom filtering logic into process_file when loading
DAGs from dag_folder.

For that I need the file path and the DAG object that contains owner and
name. Based on these 3 pieces of information, I want to decide whether to
accept the DAG or not.

It is possible to modify airflow/models.py to add this custom logic, but
I'm looking for a way to accomplish that via configuration vs. code change.
Are there any suggestions how to address this (maybe by adding a plug-in or
a configurable expression)?

Thanks,
Thomas

Re: Filtering DAGs in process_file

[Maxime Beauchemin <ma...@gmail.com>]

If in your install you have some sort of global config file
(airflow_config.py), you may be able to monkey patch the main classes in
from context, not that I would recommend that. It can be reasonable to have
your internal fork and minimize how far it goes from the trunk.

Max

On Fri, Apr 7, 2017 at 11:04 AM, Thomas Weise <th...@apache.org> wrote:

> Yes, that is possible but I still need the hook to ensure the DAG class is
> overridden.
>
> Any suggestion how this can be done without forking the code?
>
> Thanks,
> Thomas
>
>
> On Thu, Apr 6, 2017 at 6:02 PM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
> > You could derive the DAG class and override process_file. Then you'd need
> > to make sure your users/pipelines import your derivative instead of
> > `airflow.DAG`.
> >
> > Max
> >
> > On Thu, Apr 6, 2017 at 5:41 PM, Thomas Weise <th...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > I want to include custom filtering logic into process_file when loading
> > > DAGs from dag_folder.
> > >
> > > For that I need the file path and the DAG object that contains owner
> and
> > > name. Based on these 3 pieces of information, I want to decide whether
> to
> > > accept the DAG or not.
> > >
> > > It is possible to modify airflow/models.py to add this custom logic,
> but
> > > I'm looking for a way to accomplish that via configuration vs. code
> > change.
> > > Are there any suggestions how to address this (maybe by adding a
> plug-in
> > or
> > > a configurable expression)?
> > >
> > > Thanks,
> > > Thomas
> > >
> >
>

Re: Filtering DAGs in process_file

[Thomas Weise <th...@apache.org>]

Yes, that is possible but I still need the hook to ensure the DAG class is
overridden.

Any suggestion how this can be done without forking the code?

Thanks,
Thomas


On Thu, Apr 6, 2017 at 6:02 PM, Maxime Beauchemin <
maximebeauchemin@gmail.com> wrote:

> You could derive the DAG class and override process_file. Then you'd need
> to make sure your users/pipelines import your derivative instead of
> `airflow.DAG`.
>
> Max
>
> On Thu, Apr 6, 2017 at 5:41 PM, Thomas Weise <th...@apache.org> wrote:
>
> > Hi,
> >
> > I want to include custom filtering logic into process_file when loading
> > DAGs from dag_folder.
> >
> > For that I need the file path and the DAG object that contains owner and
> > name. Based on these 3 pieces of information, I want to decide whether to
> > accept the DAG or not.
> >
> > It is possible to modify airflow/models.py to add this custom logic, but
> > I'm looking for a way to accomplish that via configuration vs. code
> change.
> > Are there any suggestions how to address this (maybe by adding a plug-in
> or
> > a configurable expression)?
> >
> > Thanks,
> > Thomas
> >
>

Re: Filtering DAGs in process_file

[Maxime Beauchemin <ma...@gmail.com>]

You could derive the DAG class and override process_file. Then you'd need
to make sure your users/pipelines import your derivative instead of
`airflow.DAG`.

Max

On Thu, Apr 6, 2017 at 5:41 PM, Thomas Weise <th...@apache.org> wrote:

> Hi,
>
> I want to include custom filtering logic into process_file when loading
> DAGs from dag_folder.
>
> For that I need the file path and the DAG object that contains owner and
> name. Based on these 3 pieces of information, I want to decide whether to
> accept the DAG or not.
>
> It is possible to modify airflow/models.py to add this custom logic, but
> I'm looking for a way to accomplish that via configuration vs. code change.
> Are there any suggestions how to address this (maybe by adding a plug-in or
> a configurable expression)?
>
> Thanks,
> Thomas
>