You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <Ja...@polidea.com> on 2020/02/18 12:33:17 UTC
[DISCUSS] Airflow's extensibility options ?
I have another discussion to start. We've recently talked to a number of
customers who are extending airflow. It's often the case that people are
modifying airflow's source code and later have a hard time with updating it
when newer versions of Airflow are released.
This is a common trait - we've heard the same story from at least three of
our customers and it was also mentioned today at Slack by one of the users:
https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 and
I heard some anecdotal evidence of people doing it at many events I spoke.
We have a plugin mechanism that allows for some extensibility but I wonder
if we should do something more complete. Maybe we could gather from people
a list of ways people are extending Airflow currently by modifying it's
code and maybe we can come up with some "extension points" that we might
introduce to Airflow to let them add custom functionality they want rather
than modifying Airflow's code.
I think many of the extensions do not need to modify Airflow's source code,
and there won't be many extensions (maybe even we already have all that we
need but people do not know that they can extend airflow without modifying
the code. I think it would require a bit more description (and maybe some
verification) of what internal API of Airflow is for those extensions so
that we can keep backwards compatibility
Let me know what you think? Is it worth it? Do you see any problems with
trying to manage that? Maybe that's something we could introduce in 2.0 (at
least by better documenting what is "an extension" and providing some
examples on how Airflow can be extended.
J.
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Airflow's extensibility options ?
Posted by Ash Berlin-Taylor <as...@apache.org>.
Yes I've done some fairly horrible things to Airflow in a plugin (horrible only in how I did it - using functools.wrap) to:
- create custom db tables in upgrade db func
- start an extra background thread when the scheduler started.
But I did it using wraps/"monkey-patching" precisely because I couldn't think of a proper way of doing it.
-a
On 23 February 2020 08:36:02 GMT, Jarek Potiuk <Ja...@polidea.com> wrote:
>> The main extension point that is missed by us is DagBag. Its current
>implementation is multi-layered, and it looks like it's starting to be
>the main point of interest around the community.
>>
>> DagBag affects user deployments in the following ways:
>> * being able to read dags from remote FS/DB
>> * being able to read dags from different sources (starting from
>multiple folders..)
>> * being able to read specific version of the DAG (used by UI, now
>we have optimization around the latest version of DAG coming from DB,
>but UI still can't see the old version))
>> * being able effectively to generate DAGs without putting the code
>into dag.py
>
>I think next week we have the call about dag serialisation
>https://lists.apache.org/thread.html/rb3b89fb90709782c78fdbd2c181a7df4acc34c8ff7ed997e4b9792a7%40%3Cdev.airflow.apache.org%3E
>- so part of this could be interesting. But It looks to me that we
>need to restart the discussion on DAG fetching. It was a heated
>discussion some year ago, but seems there is a renewed interest for it
>in the community. Maybe someone would like to take a lead on it and
>start a separate discussions on it?
>
>Any other reasons people are modifying Airflow's source code to get
>what Airflow does not give them?
>
>J.
>>
>> The workaround for this is in the monkey-patching airflow code.
>However, that's not a solution, defining this extension as a plugin can
>help a lot!
>> DagBag is a one-purpose element, the API around it can be easily
>defined.
>>
>> Another area is hooks around Operator execution. I'll leave it for
>another day :)
>>
>> Evgeny.
>> databand.ai
>>
>> On 2020/02/18 12:33:17, Jarek Potiuk <Ja...@polidea.com>
>wrote:
>> > I have another discussion to start. We've recently talked to a
>number of
>> > customers who are extending airflow. It's often the case that
>people are
>> > modifying airflow's source code and later have a hard time with
>updating it
>> > when newer versions of Airflow are released.
>> >
>> > This is a common trait - we've heard the same story from at least
>three of
>> > our customers and it was also mentioned today at Slack by one of
>the users:
>> >
>https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800
>and
>> > I heard some anecdotal evidence of people doing it at many events I
>spoke.
>> >
>> > We have a plugin mechanism that allows for some extensibility but I
>wonder
>> > if we should do something more complete. Maybe we could gather from
>people
>> > a list of ways people are extending Airflow currently by modifying
>it's
>> > code and maybe we can come up with some "extension points" that we
>might
>> > introduce to Airflow to let them add custom functionality they want
>rather
>> > than modifying Airflow's code.
>> >
>> > I think many of the extensions do not need to modify Airflow's
>source code,
>> > and there won't be many extensions (maybe even we already have all
>that we
>> > need but people do not know that they can extend airflow without
>modifying
>> > the code. I think it would require a bit more description (and
>maybe some
>> > verification) of what internal API of Airflow is for those
>extensions so
>> > that we can keep backwards compatibility
>> >
>> > Let me know what you think? Is it worth it? Do you see any problems
>with
>> > trying to manage that? Maybe that's something we could introduce in
>2.0 (at
>> > least by better documenting what is "an extension" and providing
>some
>> > examples on how Airflow can be extended.
>> >
>> > J.
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> >
>> > M: +48 660 796 129 <+48660796129>
>> > [image: Polidea] <https://www.polidea.com/>
>> >
>
>
>
>--
>
>Jarek Potiuk
>Polidea | Principal Software Engineer
>
>M: +48 660 796 129
Re: [DISCUSS] Airflow's extensibility options ?
Posted by Jarek Potiuk <Ja...@polidea.com>.
> The main extension point that is missed by us is DagBag. Its current implementation is multi-layered, and it looks like it's starting to be the main point of interest around the community.
>
> DagBag affects user deployments in the following ways:
> * being able to read dags from remote FS/DB
> * being able to read dags from different sources (starting from multiple folders..)
> * being able to read specific version of the DAG (used by UI, now we have optimization around the latest version of DAG coming from DB, but UI still can't see the old version))
> * being able effectively to generate DAGs without putting the code into dag.py
I think next week we have the call about dag serialisation
https://lists.apache.org/thread.html/rb3b89fb90709782c78fdbd2c181a7df4acc34c8ff7ed997e4b9792a7%40%3Cdev.airflow.apache.org%3E
- so part of this could be interesting. But It looks to me that we
need to restart the discussion on DAG fetching. It was a heated
discussion some year ago, but seems there is a renewed interest for it
in the community. Maybe someone would like to take a lead on it and
start a separate discussions on it?
Any other reasons people are modifying Airflow's source code to get
what Airflow does not give them?
J.
>
> The workaround for this is in the monkey-patching airflow code. However, that's not a solution, defining this extension as a plugin can help a lot!
> DagBag is a one-purpose element, the API around it can be easily defined.
>
> Another area is hooks around Operator execution. I'll leave it for another day :)
>
> Evgeny.
> databand.ai
>
> On 2020/02/18 12:33:17, Jarek Potiuk <Ja...@polidea.com> wrote:
> > I have another discussion to start. We've recently talked to a number of
> > customers who are extending airflow. It's often the case that people are
> > modifying airflow's source code and later have a hard time with updating it
> > when newer versions of Airflow are released.
> >
> > This is a common trait - we've heard the same story from at least three of
> > our customers and it was also mentioned today at Slack by one of the users:
> > https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 and
> > I heard some anecdotal evidence of people doing it at many events I spoke.
> >
> > We have a plugin mechanism that allows for some extensibility but I wonder
> > if we should do something more complete. Maybe we could gather from people
> > a list of ways people are extending Airflow currently by modifying it's
> > code and maybe we can come up with some "extension points" that we might
> > introduce to Airflow to let them add custom functionality they want rather
> > than modifying Airflow's code.
> >
> > I think many of the extensions do not need to modify Airflow's source code,
> > and there won't be many extensions (maybe even we already have all that we
> > need but people do not know that they can extend airflow without modifying
> > the code. I think it would require a bit more description (and maybe some
> > verification) of what internal API of Airflow is for those extensions so
> > that we can keep backwards compatibility
> >
> > Let me know what you think? Is it worth it? Do you see any problems with
> > trying to manage that? Maybe that's something we could introduce in 2.0 (at
> > least by better documenting what is "an extension" and providing some
> > examples on how Airflow can be extended.
> >
> > J.
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: [DISCUSS] Airflow's extensibility options ?
Posted by Evgeny Shulman <ev...@databand.ai>.
Hi Jarek
It's a great topic to start with.
The main extension point that is missed by us is DagBag. Its current implementation is multi-layered, and it looks like it's starting to be the main point of interest around the community.
DagBag affects user deployments in the following ways:
* being able to read dags from remote FS/DB
* being able to read dags from different sources (starting from multiple folders..)
* being able to read specific version of the DAG (used by UI, now we have optimization around the latest version of DAG coming from DB, but UI still can't see the old version))
* being able effectively to generate DAGs without putting the code into dag.py
The workaround for this is in the monkey-patching airflow code. However, that's not a solution, defining this extension as a plugin can help a lot!
DagBag is a one-purpose element, the API around it can be easily defined.
Another area is hooks around Operator execution. I'll leave it for another day :)
Evgeny.
databand.ai
On 2020/02/18 12:33:17, Jarek Potiuk <Ja...@polidea.com> wrote:
> I have another discussion to start. We've recently talked to a number of
> customers who are extending airflow. It's often the case that people are
> modifying airflow's source code and later have a hard time with updating it
> when newer versions of Airflow are released.
>
> This is a common trait - we've heard the same story from at least three of
> our customers and it was also mentioned today at Slack by one of the users:
> https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 and
> I heard some anecdotal evidence of people doing it at many events I spoke.
>
> We have a plugin mechanism that allows for some extensibility but I wonder
> if we should do something more complete. Maybe we could gather from people
> a list of ways people are extending Airflow currently by modifying it's
> code and maybe we can come up with some "extension points" that we might
> introduce to Airflow to let them add custom functionality they want rather
> than modifying Airflow's code.
>
> I think many of the extensions do not need to modify Airflow's source code,
> and there won't be many extensions (maybe even we already have all that we
> need but people do not know that they can extend airflow without modifying
> the code. I think it would require a bit more description (and maybe some
> verification) of what internal API of Airflow is for those extensions so
> that we can keep backwards compatibility
>
> Let me know what you think? Is it worth it? Do you see any problems with
> trying to manage that? Maybe that's something we could introduce in 2.0 (at
> least by better documenting what is "an extension" and providing some
> examples on how Airflow can be extended.
>
> J.
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
Re: [DISCUSS] Airflow's extensibility options ?
Posted by Jarek Potiuk <Ja...@polidea.com>.
I think maybe we should simply gather some evidence on what users are using
the extensions for. I can put together some evidence from my customers, and
maybe others could contribute theirs?
Do others have experience with similar case where people extend Airflow by
modifying Airflow's source code?
I agree that the more flexible we are the more complex maintenance and
backwards compatibility is so it all should be done very, very carefully.
J.
On Tue, Feb 18, 2020 at 6:13 PM Dan Davydov <dd...@twitter.com.invalid>
wrote:
> I think the kinds of plugins you are talking make sense in some contexts
> (e.g. custom views when clicking on a task in the UI, e.g. ability to
> visualize the data an ETL job provides) but we should be careful allowing
> extensions to more core parts, it will become very hard to change/maintain
> the project. From what I've seen the biggest gaps in customization/lowest
> hanging fruit is extending the REST API and making it more robust instead
> of experimental.
>
> On Tue, Feb 18, 2020 at 7:33 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > I have another discussion to start. We've recently talked to a number of
> > customers who are extending airflow. It's often the case that people are
> > modifying airflow's source code and later have a hard time with updating
> it
> > when newer versions of Airflow are released.
> >
> > This is a common trait - we've heard the same story from at least three
> of
> > our customers and it was also mentioned today at Slack by one of the
> users:
> > https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800
> and
> > I heard some anecdotal evidence of people doing it at many events I
> spoke.
> >
> > We have a plugin mechanism that allows for some extensibility but I
> wonder
> > if we should do something more complete. Maybe we could gather from
> people
> > a list of ways people are extending Airflow currently by modifying it's
> > code and maybe we can come up with some "extension points" that we might
> > introduce to Airflow to let them add custom functionality they want
> rather
> > than modifying Airflow's code.
> >
> > I think many of the extensions do not need to modify Airflow's source
> code,
> > and there won't be many extensions (maybe even we already have all that
> we
> > need but people do not know that they can extend airflow without
> modifying
> > the code. I think it would require a bit more description (and maybe some
> > verification) of what internal API of Airflow is for those extensions so
> > that we can keep backwards compatibility
> >
> > Let me know what you think? Is it worth it? Do you see any problems with
> > trying to manage that? Maybe that's something we could introduce in 2.0
> (at
> > least by better documenting what is "an extension" and providing some
> > examples on how Airflow can be extended.
> >
> > J.
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [DISCUSS] Airflow's extensibility options ?
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
I think the kinds of plugins you are talking make sense in some contexts
(e.g. custom views when clicking on a task in the UI, e.g. ability to
visualize the data an ETL job provides) but we should be careful allowing
extensions to more core parts, it will become very hard to change/maintain
the project. From what I've seen the biggest gaps in customization/lowest
hanging fruit is extending the REST API and making it more robust instead
of experimental.
On Tue, Feb 18, 2020 at 7:33 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> I have another discussion to start. We've recently talked to a number of
> customers who are extending airflow. It's often the case that people are
> modifying airflow's source code and later have a hard time with updating it
> when newer versions of Airflow are released.
>
> This is a common trait - we've heard the same story from at least three of
> our customers and it was also mentioned today at Slack by one of the users:
> https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 and
> I heard some anecdotal evidence of people doing it at many events I spoke.
>
> We have a plugin mechanism that allows for some extensibility but I wonder
> if we should do something more complete. Maybe we could gather from people
> a list of ways people are extending Airflow currently by modifying it's
> code and maybe we can come up with some "extension points" that we might
> introduce to Airflow to let them add custom functionality they want rather
> than modifying Airflow's code.
>
> I think many of the extensions do not need to modify Airflow's source code,
> and there won't be many extensions (maybe even we already have all that we
> need but people do not know that they can extend airflow without modifying
> the code. I think it would require a bit more description (and maybe some
> verification) of what internal API of Airflow is for those extensions so
> that we can keep backwards compatibility
>
> Let me know what you think? Is it worth it? Do you see any problems with
> trying to manage that? Maybe that's something we could introduce in 2.0 (at
> least by better documenting what is "an extension" and providing some
> examples on how Airflow can be extended.
>
> J.
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>