You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Gerard Casas Saez <gc...@twitter.com.INVALID> on 2020/01/28 20:56:14 UTC
[DISCUSS] Airflow functional DAGs
Hi everyone!
Starting a conversation here about extending Airflow for supporting a more functional way to define DAGs including better data dependency/lineage clarity on the DAG itself. I believe adding this functional extension would allow to support more Data pipelines use cases and extend Airflow usage, while being backwards compatible.
My proposal is to include a __call__ function that would dynamically replace class attributes before executing the `pre_execute` and `execute` function. This tied with a XComArg, a class that points to a previous task XCom pushed value, allows to define DAGs in a more functional and data approach. Basically my proposal is:
• Add a __call__ function in BaseOperator that accepts Inlets (in my case its XComArgs)
• Add a simple Python Decorator to create PythonOperators easily from functions
• Resolves them before executing any main operator execution on a worker/K8s pod….
• Log their values on execution time and make them visible in the Web UI for each task instance
• Set attribute in the operator instance
• Executes the operator and returns an XComArgs that can later be tied in a new operator as an Inlet…
I wrote a small extension (~200 lines of code) that implements it. Here’s what it would look like with existing operators (and that works with some simple tweaks on the already existing codebase):
get_ip = SimpleHttpOperator(
task_id='get_ip', endpoint='get', method='GET', xcom_push=True
)
@operation
def email_subject_generator(raw_json: str) -> str:
external_ip = json.loads(raw_json)['origin']
return f"Server connected from {external_ip}”
@operation
def email_body_generator(raw_json: str) -> str:
return """
Seems like today your server executing Airflow is connected from the external IP {origin}<br>
<br>
Have a nice day!
""".format(**json.loads(raw_json))
send_email = EmailOperator(
task_id='send_email',
to="example@example.com",
subject='',
html_content=''
)
ip_info = get_ip()
subject = email_subject_generator(ip_info)
body = email_body_generator(ip_info)
send_email(subject=subject, html_content=body)
As someone wise sometime said, code is better than words, so here’s my experimental code: https://github.com/casassg/corrent and a couple of example DAGs I wrote: https://github.com/casassg/corrent/tree/master/dags
This approach is highly inspired from some alternatives that have lately appeared on ETL pipelines world (ex: Dagster) that support more functional ways of defining DAGs.
I believe this approach could benefit the Airflow community to grow. What do you think? Any thoughts on functional DAGs? Would love to hear what are pros and cons of this expansion. To me, it seems highly linked to the Lineage discussion happening at the same time, as both discussions are around improving data pipelines.
Thanks!
Gerard Casas Saez
Twitter | Cortex | @casassaez
Re: [DISCUSS] Airflow functional DAGs
Posted by Gerard Casas Saez <gc...@twitter.com.INVALID>.
A bit of an update: discussing some design aspects w Tomasz in the doc.
My plan is to transition this into a Confluence AIP next week (Thursday) and send a separate note to the mailing list. This should allow people to chip in with their thoughts before is more established and iterate on it meanwhile. And also allow for more visibility next week once published on Confluence.
Let me know if this seems appropriate or not :D
Gerard Casas Saez
Twitter | Cortex | @casassaez
On Feb 5, 2020, 2:46 PM -0700, Gerard Casas Saez <gc...@twitter.com>, wrote:
> Wrote this small proposal https://docs.google.com/document/d/1ammZ2iGmGuoXNCtbpLdnj2D23nEYrCaLMbETjim3C0Q/edit?usp=sharing I can add it to Confluence as well if pointed to where should it live (sorry not really familiar enough w Airflow Confluence).
>
> Working to add some more context on existing frameworks that inspired the proposal.
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez
> On Feb 5, 2020, 2:36 PM -0700, Dan Davydov <dd...@twitter.com.invalid>, wrote:
> > Ok let's try it, don't know if we're violating some Apache process here but
> > I guess we'll find out :).
> >
> > On Wed, Feb 5, 2020 at 4:22 PM Tomasz Urbaszek <to...@polidea.com>
> > wrote:
> >
> > > Google docs is good to work out final version that can be published on
> > > confluence. But that’s only my opinion.
> > >
> > > T.
> > >
> > > On Wed, 5 Feb 2020 at 22:12, Dan Davydov <dd...@twitter.com.invalid>
> > > wrote:
> > >
> > > > Traditionally we've done this in confluence within the AIP although I
> > > think
> > > > I would prefer google docs at some point in the future maybe :). I would
> > > > use confluence though for this.
> > > >
> > > > On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez
> > > > <gc...@twitter.com.invalid> wrote:
> > > >
> > > > > Happy to drive this. What would be a good place to put this design doc?
> > > > > Guessing confluence, not sure under what directory though.
> > > > >
> > > > > Gerard Casas Saez
> > > > > Twitter | Cortex | @casassaez
> > > > > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <Jarek.Potiuk@polidea.com
> > > > ,
> > > > > wrote:
> > > > > > +1 short design doc would be cool.
> > > > > >
> > > > > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
> > > > > > tomasz.urbaszek@polidea.com> napisał:
> > > > > >
> > > > > > > Do you think we should start with some design doc for that? In this
> > > > > > > way, we can work out the best solution and allow other to add 2
> > > > cents?
> > > > > > >
> > > > > > > T.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> > > > > > > <da...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > I think if we’re not breaking any other operators (which I doubt
> > > we
> > > > > are)
> > > > > > > it’s a great 2.0 feature. It would also look great in a “What’s New
> > > > in
> > > > > > > Airflow 2.0” announcement ;).
> > > > > > > >
> > > > > > > > Docs are always a challenge, but we could set up a google doc and
> > > > > hack
> > > > > > > it out in a day or two.
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > via Newton Mail [
> > > > > > >
> > > > >
> > > >
> > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > > > > > ]
> > > > > > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com>
> > > > > > > wrote:
> > > > > > > > I like the idea, especially the backwards compatibility.
> > > > > > > >
> > > > > > > > I would love to understand more about whether it will work (it
> > > > looks
> > > > > like
> > > > > > > > it will) without modifying the 100s of operators we already have.
> > > > If
> > > > > so,
> > > > > > > > this looks like a nice addition to the current way how we define
> > > > > Dags and
> > > > > > > > even allows for incremental migration from the "traditional" to
> > > > > > > > "functional" Dag definition pattern. It does not enforce it but
> > > it
> > > > > opens
> > > > > > > up
> > > > > > > > new possibilities without changing basic paradigms of Airflow.
> > > > > > > >
> > > > > > > > It looks like we could even make it available in 2.0 as there are
> > > > > hardly
> > > > > > > > any dependencies and very low risk with introducing such change.
> > > I
> > > > > think
> > > > > > > > the biggest challenge will be to write good documentation and
> > > > making
> > > > > sure
> > > > > > > > that examples are there - but maybe we could even somewhat
> > > automate
> > > > > it
> > > > > > > and
> > > > > > > > generate some part of the "functional variants" for the examples
> > > we
> > > > > have?
> > > > > > > >
> > > > > > > > WDYT Dan, others ?
> > > > > > > >
> > > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Tomasz Urbaszek
> > > > > > > Polidea | Software Engineer
> > > > > > >
> > > > > > > M: +48 505 628 493
> > > > > > > E: tomasz.urbaszek@polidea.com
> > > > > > >
> > > > > > > Unique Tech
> > > > > > > Check out our projects!
> > > > > > >
> > > > >
> > > >
> > > --
> > >
> > > Tomasz Urbaszek
> > > Polidea <https://www.polidea.com/> | Software Engineer
> > >
> > > M: +48 505 628 493 <+48505628493>
> > > E: tomasz.urbaszek@polidea.com <to...@polidea.com>
> > >
> > > Unique Tech
> > > Check out our projects! <https://www.polidea.com/our-work>
> > >
Re: [DISCUSS] Airflow functional DAGs
Posted by Gerard Casas Saez <gc...@twitter.com.INVALID>.
Wrote this small proposal https://docs.google.com/document/d/1ammZ2iGmGuoXNCtbpLdnj2D23nEYrCaLMbETjim3C0Q/edit?usp=sharing I can add it to Confluence as well if pointed to where should it live (sorry not really familiar enough w Airflow Confluence).
Working to add some more context on existing frameworks that inspired the proposal.
Gerard Casas Saez
Twitter | Cortex | @casassaez
On Feb 5, 2020, 2:36 PM -0700, Dan Davydov <dd...@twitter.com.invalid>, wrote:
> Ok let's try it, don't know if we're violating some Apache process here but
> I guess we'll find out :).
>
> On Wed, Feb 5, 2020 at 4:22 PM Tomasz Urbaszek <to...@polidea.com>
> wrote:
>
> > Google docs is good to work out final version that can be published on
> > confluence. But that’s only my opinion.
> >
> > T.
> >
> > On Wed, 5 Feb 2020 at 22:12, Dan Davydov <dd...@twitter.com.invalid>
> > wrote:
> >
> > > Traditionally we've done this in confluence within the AIP although I
> > think
> > > I would prefer google docs at some point in the future maybe :). I would
> > > use confluence though for this.
> > >
> > > On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez
> > > <gc...@twitter.com.invalid> wrote:
> > >
> > > > Happy to drive this. What would be a good place to put this design doc?
> > > > Guessing confluence, not sure under what directory though.
> > > >
> > > > Gerard Casas Saez
> > > > Twitter | Cortex | @casassaez
> > > > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <Jarek.Potiuk@polidea.com
> > > ,
> > > > wrote:
> > > > > +1 short design doc would be cool.
> > > > >
> > > > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
> > > > > tomasz.urbaszek@polidea.com> napisał:
> > > > >
> > > > > > Do you think we should start with some design doc for that? In this
> > > > > > way, we can work out the best solution and allow other to add 2
> > > cents?
> > > > > >
> > > > > > T.
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> > > > > > <da...@gmail.com> wrote:
> > > > > > >
> > > > > > > I think if we’re not breaking any other operators (which I doubt
> > we
> > > > are)
> > > > > > it’s a great 2.0 feature. It would also look great in a “What’s New
> > > in
> > > > > > Airflow 2.0” announcement ;).
> > > > > > >
> > > > > > > Docs are always a challenge, but we could set up a google doc and
> > > > hack
> > > > > > it out in a day or two.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > via Newton Mail [
> > > > > >
> > > >
> > >
> > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > > > > ]
> > > > > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com>
> > > > > > wrote:
> > > > > > > I like the idea, especially the backwards compatibility.
> > > > > > >
> > > > > > > I would love to understand more about whether it will work (it
> > > looks
> > > > like
> > > > > > > it will) without modifying the 100s of operators we already have.
> > > If
> > > > so,
> > > > > > > this looks like a nice addition to the current way how we define
> > > > Dags and
> > > > > > > even allows for incremental migration from the "traditional" to
> > > > > > > "functional" Dag definition pattern. It does not enforce it but
> > it
> > > > opens
> > > > > > up
> > > > > > > new possibilities without changing basic paradigms of Airflow.
> > > > > > >
> > > > > > > It looks like we could even make it available in 2.0 as there are
> > > > hardly
> > > > > > > any dependencies and very low risk with introducing such change.
> > I
> > > > think
> > > > > > > the biggest challenge will be to write good documentation and
> > > making
> > > > sure
> > > > > > > that examples are there - but maybe we could even somewhat
> > automate
> > > > it
> > > > > > and
> > > > > > > generate some part of the "functional variants" for the examples
> > we
> > > > have?
> > > > > > >
> > > > > > > WDYT Dan, others ?
> > > > > > >
> > > > > > > J.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Tomasz Urbaszek
> > > > > > Polidea | Software Engineer
> > > > > >
> > > > > > M: +48 505 628 493
> > > > > > E: tomasz.urbaszek@polidea.com
> > > > > >
> > > > > > Unique Tech
> > > > > > Check out our projects!
> > > > > >
> > > >
> > >
> > --
> >
> > Tomasz Urbaszek
> > Polidea <https://www.polidea.com/> | Software Engineer
> >
> > M: +48 505 628 493 <+48505628493>
> > E: tomasz.urbaszek@polidea.com <to...@polidea.com>
> >
> > Unique Tech
> > Check out our projects! <https://www.polidea.com/our-work>
> >
Re: [DISCUSS] Airflow functional DAGs
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
Ok let's try it, don't know if we're violating some Apache process here but
I guess we'll find out :).
On Wed, Feb 5, 2020 at 4:22 PM Tomasz Urbaszek <to...@polidea.com>
wrote:
> Google docs is good to work out final version that can be published on
> confluence. But that’s only my opinion.
>
> T.
>
> On Wed, 5 Feb 2020 at 22:12, Dan Davydov <dd...@twitter.com.invalid>
> wrote:
>
> > Traditionally we've done this in confluence within the AIP although I
> think
> > I would prefer google docs at some point in the future maybe :). I would
> > use confluence though for this.
> >
> > On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez
> > <gc...@twitter.com.invalid> wrote:
> >
> > > Happy to drive this. What would be a good place to put this design doc?
> > > Guessing confluence, not sure under what directory though.
> > >
> > > Gerard Casas Saez
> > > Twitter | Cortex | @casassaez
> > > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >,
> > > wrote:
> > > > +1 short design doc would be cool.
> > > >
> > > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
> > > > tomasz.urbaszek@polidea.com> napisał:
> > > >
> > > > > Do you think we should start with some design doc for that? In this
> > > > > way, we can work out the best solution and allow other to add 2
> > cents?
> > > > >
> > > > > T.
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> > > > > <da...@gmail.com> wrote:
> > > > > >
> > > > > > I think if we’re not breaking any other operators (which I doubt
> we
> > > are)
> > > > > it’s a great 2.0 feature. It would also look great in a “What’s New
> > in
> > > > > Airflow 2.0” announcement ;).
> > > > > >
> > > > > > Docs are always a challenge, but we could set up a google doc and
> > > hack
> > > > > it out in a day or two.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > via Newton Mail [
> > > > >
> > >
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > > > ]
> > > > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > wrote:
> > > > > > I like the idea, especially the backwards compatibility.
> > > > > >
> > > > > > I would love to understand more about whether it will work (it
> > looks
> > > like
> > > > > > it will) without modifying the 100s of operators we already have.
> > If
> > > so,
> > > > > > this looks like a nice addition to the current way how we define
> > > Dags and
> > > > > > even allows for incremental migration from the "traditional" to
> > > > > > "functional" Dag definition pattern. It does not enforce it but
> it
> > > opens
> > > > > up
> > > > > > new possibilities without changing basic paradigms of Airflow.
> > > > > >
> > > > > > It looks like we could even make it available in 2.0 as there are
> > > hardly
> > > > > > any dependencies and very low risk with introducing such change.
> I
> > > think
> > > > > > the biggest challenge will be to write good documentation and
> > making
> > > sure
> > > > > > that examples are there - but maybe we could even somewhat
> automate
> > > it
> > > > > and
> > > > > > generate some part of the "functional variants" for the examples
> we
> > > have?
> > > > > >
> > > > > > WDYT Dan, others ?
> > > > > >
> > > > > > J.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Tomasz Urbaszek
> > > > > Polidea | Software Engineer
> > > > >
> > > > > M: +48 505 628 493
> > > > > E: tomasz.urbaszek@polidea.com
> > > > >
> > > > > Unique Tech
> > > > > Check out our projects!
> > > > >
> > >
> >
> --
>
> Tomasz Urbaszek
> Polidea <https://www.polidea.com/> | Software Engineer
>
> M: +48 505 628 493 <+48505628493>
> E: tomasz.urbaszek@polidea.com <to...@polidea.com>
>
> Unique Tech
> Check out our projects! <https://www.polidea.com/our-work>
>
Re: [DISCUSS] Airflow functional DAGs
Posted by Tomasz Urbaszek <to...@polidea.com>.
Google docs is good to work out final version that can be published on
confluence. But that’s only my opinion.
T.
On Wed, 5 Feb 2020 at 22:12, Dan Davydov <dd...@twitter.com.invalid>
wrote:
> Traditionally we've done this in confluence within the AIP although I think
> I would prefer google docs at some point in the future maybe :). I would
> use confluence though for this.
>
> On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez
> <gc...@twitter.com.invalid> wrote:
>
> > Happy to drive this. What would be a good place to put this design doc?
> > Guessing confluence, not sure under what directory though.
> >
> > Gerard Casas Saez
> > Twitter | Cortex | @casassaez
> > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <Ja...@polidea.com>,
> > wrote:
> > > +1 short design doc would be cool.
> > >
> > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
> > > tomasz.urbaszek@polidea.com> napisał:
> > >
> > > > Do you think we should start with some design doc for that? In this
> > > > way, we can work out the best solution and allow other to add 2
> cents?
> > > >
> > > > T.
> > > >
> > > >
> > > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> > > > <da...@gmail.com> wrote:
> > > > >
> > > > > I think if we’re not breaking any other operators (which I doubt we
> > are)
> > > > it’s a great 2.0 feature. It would also look great in a “What’s New
> in
> > > > Airflow 2.0” announcement ;).
> > > > >
> > > > > Docs are always a challenge, but we could set up a google doc and
> > hack
> > > > it out in a day or two.
> > > > >
> > > > > +1
> > > > >
> > > > > via Newton Mail [
> > > >
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > > ]
> > > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > > > I like the idea, especially the backwards compatibility.
> > > > >
> > > > > I would love to understand more about whether it will work (it
> looks
> > like
> > > > > it will) without modifying the 100s of operators we already have.
> If
> > so,
> > > > > this looks like a nice addition to the current way how we define
> > Dags and
> > > > > even allows for incremental migration from the "traditional" to
> > > > > "functional" Dag definition pattern. It does not enforce it but it
> > opens
> > > > up
> > > > > new possibilities without changing basic paradigms of Airflow.
> > > > >
> > > > > It looks like we could even make it available in 2.0 as there are
> > hardly
> > > > > any dependencies and very low risk with introducing such change. I
> > think
> > > > > the biggest challenge will be to write good documentation and
> making
> > sure
> > > > > that examples are there - but maybe we could even somewhat automate
> > it
> > > > and
> > > > > generate some part of the "functional variants" for the examples we
> > have?
> > > > >
> > > > > WDYT Dan, others ?
> > > > >
> > > > > J.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Tomasz Urbaszek
> > > > Polidea | Software Engineer
> > > >
> > > > M: +48 505 628 493
> > > > E: tomasz.urbaszek@polidea.com
> > > >
> > > > Unique Tech
> > > > Check out our projects!
> > > >
> >
>
--
Tomasz Urbaszek
Polidea <https://www.polidea.com/> | Software Engineer
M: +48 505 628 493 <+48505628493>
E: tomasz.urbaszek@polidea.com <to...@polidea.com>
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>
Re: [DISCUSS] Airflow functional DAGs
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
Traditionally we've done this in confluence within the AIP although I think
I would prefer google docs at some point in the future maybe :). I would
use confluence though for this.
On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez
<gc...@twitter.com.invalid> wrote:
> Happy to drive this. What would be a good place to put this design doc?
> Guessing confluence, not sure under what directory though.
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez
> On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <Ja...@polidea.com>,
> wrote:
> > +1 short design doc would be cool.
> >
> > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
> > tomasz.urbaszek@polidea.com> napisał:
> >
> > > Do you think we should start with some design doc for that? In this
> > > way, we can work out the best solution and allow other to add 2 cents?
> > >
> > > T.
> > >
> > >
> > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> > > <da...@gmail.com> wrote:
> > > >
> > > > I think if we’re not breaking any other operators (which I doubt we
> are)
> > > it’s a great 2.0 feature. It would also look great in a “What’s New in
> > > Airflow 2.0” announcement ;).
> > > >
> > > > Docs are always a challenge, but we could set up a google doc and
> hack
> > > it out in a day or two.
> > > >
> > > > +1
> > > >
> > > > via Newton Mail [
> > >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > ]
> > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > > > I like the idea, especially the backwards compatibility.
> > > >
> > > > I would love to understand more about whether it will work (it looks
> like
> > > > it will) without modifying the 100s of operators we already have. If
> so,
> > > > this looks like a nice addition to the current way how we define
> Dags and
> > > > even allows for incremental migration from the "traditional" to
> > > > "functional" Dag definition pattern. It does not enforce it but it
> opens
> > > up
> > > > new possibilities without changing basic paradigms of Airflow.
> > > >
> > > > It looks like we could even make it available in 2.0 as there are
> hardly
> > > > any dependencies and very low risk with introducing such change. I
> think
> > > > the biggest challenge will be to write good documentation and making
> sure
> > > > that examples are there - but maybe we could even somewhat automate
> it
> > > and
> > > > generate some part of the "functional variants" for the examples we
> have?
> > > >
> > > > WDYT Dan, others ?
> > > >
> > > > J.
> > >
> > >
> > >
> > > --
> > >
> > > Tomasz Urbaszek
> > > Polidea | Software Engineer
> > >
> > > M: +48 505 628 493
> > > E: tomasz.urbaszek@polidea.com
> > >
> > > Unique Tech
> > > Check out our projects!
> > >
>
Re: [DISCUSS] Airflow functional DAGs
Posted by Gerard Casas Saez <gc...@twitter.com.INVALID>.
Happy to drive this. What would be a good place to put this design doc? Guessing confluence, not sure under what directory though.
Gerard Casas Saez
Twitter | Cortex | @casassaez
On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <Ja...@polidea.com>, wrote:
> +1 short design doc would be cool.
>
> wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
> tomasz.urbaszek@polidea.com> napisał:
>
> > Do you think we should start with some design doc for that? In this
> > way, we can work out the best solution and allow other to add 2 cents?
> >
> > T.
> >
> >
> > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> > <da...@gmail.com> wrote:
> > >
> > > I think if we’re not breaking any other operators (which I doubt we are)
> > it’s a great 2.0 feature. It would also look great in a “What’s New in
> > Airflow 2.0” announcement ;).
> > >
> > > Docs are always a challenge, but we could set up a google doc and hack
> > it out in a day or two.
> > >
> > > +1
> > >
> > > via Newton Mail [
> > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > ]
> > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> > > I like the idea, especially the backwards compatibility.
> > >
> > > I would love to understand more about whether it will work (it looks like
> > > it will) without modifying the 100s of operators we already have. If so,
> > > this looks like a nice addition to the current way how we define Dags and
> > > even allows for incremental migration from the "traditional" to
> > > "functional" Dag definition pattern. It does not enforce it but it opens
> > up
> > > new possibilities without changing basic paradigms of Airflow.
> > >
> > > It looks like we could even make it available in 2.0 as there are hardly
> > > any dependencies and very low risk with introducing such change. I think
> > > the biggest challenge will be to write good documentation and making sure
> > > that examples are there - but maybe we could even somewhat automate it
> > and
> > > generate some part of the "functional variants" for the examples we have?
> > >
> > > WDYT Dan, others ?
> > >
> > > J.
> >
> >
> >
> > --
> >
> > Tomasz Urbaszek
> > Polidea | Software Engineer
> >
> > M: +48 505 628 493
> > E: tomasz.urbaszek@polidea.com
> >
> > Unique Tech
> > Check out our projects!
> >
Re: [DISCUSS] Airflow functional DAGs
Posted by Jarek Potiuk <Ja...@polidea.com>.
+1 short design doc would be cool.
wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek <
tomasz.urbaszek@polidea.com> napisał:
> Do you think we should start with some design doc for that? In this
> way, we can work out the best solution and allow other to add 2 cents?
>
> T.
>
>
> On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
> <da...@gmail.com> wrote:
> >
> > I think if we’re not breaking any other operators (which I doubt we are)
> it’s a great 2.0 feature. It would also look great in a “What’s New in
> Airflow 2.0” announcement ;).
> >
> > Docs are always a challenge, but we could set up a google doc and hack
> it out in a day or two.
> >
> > +1
> >
> > via Newton Mail [
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> ]
> > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
> > I like the idea, especially the backwards compatibility.
> >
> > I would love to understand more about whether it will work (it looks like
> > it will) without modifying the 100s of operators we already have. If so,
> > this looks like a nice addition to the current way how we define Dags and
> > even allows for incremental migration from the "traditional" to
> > "functional" Dag definition pattern. It does not enforce it but it opens
> up
> > new possibilities without changing basic paradigms of Airflow.
> >
> > It looks like we could even make it available in 2.0 as there are hardly
> > any dependencies and very low risk with introducing such change. I think
> > the biggest challenge will be to write good documentation and making sure
> > that examples are there - but maybe we could even somewhat automate it
> and
> > generate some part of the "functional variants" for the examples we have?
> >
> > WDYT Dan, others ?
> >
> > J.
>
>
>
> --
>
> Tomasz Urbaszek
> Polidea | Software Engineer
>
> M: +48 505 628 493
> E: tomasz.urbaszek@polidea.com
>
> Unique Tech
> Check out our projects!
>
Re: [DISCUSS] Airflow functional DAGs
Posted by Tomasz Urbaszek <to...@polidea.com>.
Do you think we should start with some design doc for that? In this
way, we can work out the best solution and allow other to add 2 cents?
T.
On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman
<da...@gmail.com> wrote:
>
> I think if we’re not breaking any other operators (which I doubt we are) it’s a great 2.0 feature. It would also look great in a “What’s New in Airflow 2.0” announcement ;).
>
> Docs are always a challenge, but we could set up a google doc and hack it out in a day or two.
>
> +1
>
> via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]
> On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
> I like the idea, especially the backwards compatibility.
>
> I would love to understand more about whether it will work (it looks like
> it will) without modifying the 100s of operators we already have. If so,
> this looks like a nice addition to the current way how we define Dags and
> even allows for incremental migration from the "traditional" to
> "functional" Dag definition pattern. It does not enforce it but it opens up
> new possibilities without changing basic paradigms of Airflow.
>
> It looks like we could even make it available in 2.0 as there are hardly
> any dependencies and very low risk with introducing such change. I think
> the biggest challenge will be to write good documentation and making sure
> that examples are there - but maybe we could even somewhat automate it and
> generate some part of the "functional variants" for the examples we have?
>
> WDYT Dan, others ?
>
> J.
--
Tomasz Urbaszek
Polidea | Software Engineer
M: +48 505 628 493
E: tomasz.urbaszek@polidea.com
Unique Tech
Check out our projects!
Re: [DISCUSS] Airflow functional DAGs
Posted by Daniel Imberman <da...@gmail.com>.
I think if we’re not breaking any other operators (which I doubt we are) it’s a great 2.0 feature. It would also look great in a “What’s New in Airflow 2.0” announcement ;).
Docs are always a challenge, but we could set up a google doc and hack it out in a day or two.
+1
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]
On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
I like the idea, especially the backwards compatibility.
I would love to understand more about whether it will work (it looks like
it will) without modifying the 100s of operators we already have. If so,
this looks like a nice addition to the current way how we define Dags and
even allows for incremental migration from the "traditional" to
"functional" Dag definition pattern. It does not enforce it but it opens up
new possibilities without changing basic paradigms of Airflow.
It looks like we could even make it available in 2.0 as there are hardly
any dependencies and very low risk with introducing such change. I think
the biggest challenge will be to write good documentation and making sure
that examples are there - but maybe we could even somewhat automate it and
generate some part of the "functional variants" for the examples we have?
WDYT Dan, others ?
J.
Re: [DISCUSS] Airflow functional DAGs
Posted by Jarek Potiuk <Ja...@polidea.com>.
I like the idea, especially the backwards compatibility.
I would love to understand more about whether it will work (it looks like
it will) without modifying the 100s of operators we already have. If so,
this looks like a nice addition to the current way how we define Dags and
even allows for incremental migration from the "traditional" to
"functional" Dag definition pattern. It does not enforce it but it opens up
new possibilities without changing basic paradigms of Airflow.
It looks like we could even make it available in 2.0 as there are hardly
any dependencies and very low risk with introducing such change. I think
the biggest challenge will be to write good documentation and making sure
that examples are there - but maybe we could even somewhat automate it and
generate some part of the "functional variants" for the examples we have?
WDYT Dan, others ?
J.
Re: [DISCUSS] Airflow functional DAGs
Posted by Tomasz Urbaszek <to...@polidea.com>.
+1 one for this idea. Something similar popped in my mind when I saw
Kubeflow approach some time ago.
T.
On Mon, Feb 3, 2020 at 8:05 PM Daniel Imberman
<da...@gmail.com> wrote:
>
> I like this idea a lot.
>
> We could create something similar to an “executor_config” so people can pre-populate most of the parameters necessary for a python_operator and pass it in
>
> e.g.
>
> Config = AirflowPythonConfig(…)
>
>
> @airflow.make_python_operator(config) def my_func():
>
>
> On Mon, Feb 3, 2020 at 10:36 AM, Dan Davydov <dd...@twitter.com.invalid> wrote:
> I like it : ). I think the difficulty in creating operators and chaining
> them together is one of the most common complaints about Airflow compared
> to other frameworks. Would be curious to see a comparison to other
> interfaces e.g. Dagster as well. I would be curious to see what other
> committers like Bolke think but I think the next stage is a lightweight AIP
> <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-1%3A+Improve+Airflow+Security>
> if you want to continue on this
>
> On Tue, Jan 28, 2020 at 3:56 PM Gerard Casas Saez
> <gc...@twitter.com.invalid> wrote:
>
> > Hi everyone!
> >
> > Starting a conversation here about extending Airflow for supporting a more
> > functional way to define DAGs including better data dependency/lineage
> > clarity on the DAG itself. I believe adding this functional extension would
> > allow to support more Data pipelines use cases and extend Airflow usage,
> > while being backwards compatible.
> >
> > My proposal is to include a __call__ function that would dynamically
> > replace class attributes before executing the `pre_execute` and `execute`
> > function. This tied with a XComArg, a class that points to a previous task
> > XCom pushed value, allows to define DAGs in a more functional and data
> > approach. Basically my proposal is:
> >
> >
> > • Add a __call__ function in BaseOperator that accepts Inlets (in my case
> > its XComArgs)
> > • Add a simple Python Decorator to create PythonOperators easily from
> > functions
> > • Resolves them before executing any main operator execution on a
> > worker/K8s pod….
> > • Log their values on execution time and make them visible in the Web UI
> > for each task instance
> > • Set attribute in the operator instance
> > • Executes the operator and returns an XComArgs that can later be tied in
> > a new operator as an Inlet…
> >
> >
> > I wrote a small extension (~200 lines of code) that implements it. Here’s
> > what it would look like with existing operators (and that works with some
> > simple tweaks on the already existing codebase):
> >
> > get_ip = SimpleHttpOperator(
> > task_id='get_ip', endpoint='get', method='GET', xcom_push=True
> > )
> >
> > @operation
> > def email_subject_generator(raw_json: str) -> str:
> > external_ip = json.loads(raw_json)['origin']
> > return f"Server connected from {external_ip}”
> >
> > @operation
> > def email_body_generator(raw_json: str) -> str:
> > return """
> > Seems like today your server executing Airflow is connected from the
> > external IP {origin}<br>
> > <br>
> > Have a nice day!
> > """.format(**json.loads(raw_json))
> >
> > send_email = EmailOperator(
> > task_id='send_email',
> > to="example@example.com",
> > subject='',
> > html_content=''
> > )
> >
> > ip_info = get_ip()
> > subject = email_subject_generator(ip_info)
> > body = email_body_generator(ip_info)
> > send_email(subject=subject, html_content=body)
> >
> > As someone wise sometime said, code is better than words, so here’s my
> > experimental code: https://github.com/casassg/corrent and a couple of
> > example DAGs I wrote: https://github.com/casassg/corrent/tree/master/dags
> >
> > This approach is highly inspired from some alternatives that have lately
> > appeared on ETL pipelines world (ex: Dagster) that support more functional
> > ways of defining DAGs.
> >
> > I believe this approach could benefit the Airflow community to grow. What
> > do you think? Any thoughts on functional DAGs? Would love to hear what are
> > pros and cons of this expansion. To me, it seems highly linked to the
> > Lineage discussion happening at the same time, as both discussions are
> > around improving data pipelines.
> >
> > Thanks!
> >
> > Gerard Casas Saez
> > Twitter | Cortex | @casassaez
> >
--
Tomasz Urbaszek
Polidea | Software Engineer
M: +48 505 628 493
E: tomasz.urbaszek@polidea.com
Unique Tech
Check out our projects!
Re: [DISCUSS] Airflow functional DAGs
Posted by Daniel Imberman <da...@gmail.com>.
I like this idea a lot.
We could create something similar to an “executor_config” so people can pre-populate most of the parameters necessary for a python_operator and pass it in
e.g.
Config = AirflowPythonConfig(…)
@airflow.make_python_operator(config) def my_func():
On Mon, Feb 3, 2020 at 10:36 AM, Dan Davydov <dd...@twitter.com.invalid> wrote:
I like it : ). I think the difficulty in creating operators and chaining
them together is one of the most common complaints about Airflow compared
to other frameworks. Would be curious to see a comparison to other
interfaces e.g. Dagster as well. I would be curious to see what other
committers like Bolke think but I think the next stage is a lightweight AIP
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-1%3A+Improve+Airflow+Security>
if you want to continue on this
On Tue, Jan 28, 2020 at 3:56 PM Gerard Casas Saez
<gc...@twitter.com.invalid> wrote:
> Hi everyone!
>
> Starting a conversation here about extending Airflow for supporting a more
> functional way to define DAGs including better data dependency/lineage
> clarity on the DAG itself. I believe adding this functional extension would
> allow to support more Data pipelines use cases and extend Airflow usage,
> while being backwards compatible.
>
> My proposal is to include a __call__ function that would dynamically
> replace class attributes before executing the `pre_execute` and `execute`
> function. This tied with a XComArg, a class that points to a previous task
> XCom pushed value, allows to define DAGs in a more functional and data
> approach. Basically my proposal is:
>
>
> • Add a __call__ function in BaseOperator that accepts Inlets (in my case
> its XComArgs)
> • Add a simple Python Decorator to create PythonOperators easily from
> functions
> • Resolves them before executing any main operator execution on a
> worker/K8s pod….
> • Log their values on execution time and make them visible in the Web UI
> for each task instance
> • Set attribute in the operator instance
> • Executes the operator and returns an XComArgs that can later be tied in
> a new operator as an Inlet…
>
>
> I wrote a small extension (~200 lines of code) that implements it. Here’s
> what it would look like with existing operators (and that works with some
> simple tweaks on the already existing codebase):
>
> get_ip = SimpleHttpOperator(
> task_id='get_ip', endpoint='get', method='GET', xcom_push=True
> )
>
> @operation
> def email_subject_generator(raw_json: str) -> str:
> external_ip = json.loads(raw_json)['origin']
> return f"Server connected from {external_ip}”
>
> @operation
> def email_body_generator(raw_json: str) -> str:
> return """
> Seems like today your server executing Airflow is connected from the
> external IP {origin}<br>
> <br>
> Have a nice day!
> """.format(**json.loads(raw_json))
>
> send_email = EmailOperator(
> task_id='send_email',
> to="example@example.com",
> subject='',
> html_content=''
> )
>
> ip_info = get_ip()
> subject = email_subject_generator(ip_info)
> body = email_body_generator(ip_info)
> send_email(subject=subject, html_content=body)
>
> As someone wise sometime said, code is better than words, so here’s my
> experimental code: https://github.com/casassg/corrent and a couple of
> example DAGs I wrote: https://github.com/casassg/corrent/tree/master/dags
>
> This approach is highly inspired from some alternatives that have lately
> appeared on ETL pipelines world (ex: Dagster) that support more functional
> ways of defining DAGs.
>
> I believe this approach could benefit the Airflow community to grow. What
> do you think? Any thoughts on functional DAGs? Would love to hear what are
> pros and cons of this expansion. To me, it seems highly linked to the
> Lineage discussion happening at the same time, as both discussions are
> around improving data pipelines.
>
> Thanks!
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez
>
Re: [DISCUSS] Airflow functional DAGs
Posted by Dan Davydov <dd...@twitter.com.INVALID>.
I like it : ). I think the difficulty in creating operators and chaining
them together is one of the most common complaints about Airflow compared
to other frameworks. Would be curious to see a comparison to other
interfaces e.g. Dagster as well. I would be curious to see what other
committers like Bolke think but I think the next stage is a lightweight AIP
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-1%3A+Improve+Airflow+Security>
if you want to continue on this
On Tue, Jan 28, 2020 at 3:56 PM Gerard Casas Saez
<gc...@twitter.com.invalid> wrote:
> Hi everyone!
>
> Starting a conversation here about extending Airflow for supporting a more
> functional way to define DAGs including better data dependency/lineage
> clarity on the DAG itself. I believe adding this functional extension would
> allow to support more Data pipelines use cases and extend Airflow usage,
> while being backwards compatible.
>
> My proposal is to include a __call__ function that would dynamically
> replace class attributes before executing the `pre_execute` and `execute`
> function. This tied with a XComArg, a class that points to a previous task
> XCom pushed value, allows to define DAGs in a more functional and data
> approach. Basically my proposal is:
>
>
> • Add a __call__ function in BaseOperator that accepts Inlets (in my case
> its XComArgs)
> • Add a simple Python Decorator to create PythonOperators easily from
> functions
> • Resolves them before executing any main operator execution on a
> worker/K8s pod….
> • Log their values on execution time and make them visible in the Web UI
> for each task instance
> • Set attribute in the operator instance
> • Executes the operator and returns an XComArgs that can later be tied in
> a new operator as an Inlet…
>
>
> I wrote a small extension (~200 lines of code) that implements it. Here’s
> what it would look like with existing operators (and that works with some
> simple tweaks on the already existing codebase):
>
> get_ip = SimpleHttpOperator(
> task_id='get_ip', endpoint='get', method='GET', xcom_push=True
> )
>
> @operation
> def email_subject_generator(raw_json: str) -> str:
> external_ip = json.loads(raw_json)['origin']
> return f"Server connected from {external_ip}”
>
> @operation
> def email_body_generator(raw_json: str) -> str:
> return """
> Seems like today your server executing Airflow is connected from the
> external IP {origin}<br>
> <br>
> Have a nice day!
> """.format(**json.loads(raw_json))
>
> send_email = EmailOperator(
> task_id='send_email',
> to="example@example.com",
> subject='',
> html_content=''
> )
>
> ip_info = get_ip()
> subject = email_subject_generator(ip_info)
> body = email_body_generator(ip_info)
> send_email(subject=subject, html_content=body)
>
> As someone wise sometime said, code is better than words, so here’s my
> experimental code: https://github.com/casassg/corrent and a couple of
> example DAGs I wrote: https://github.com/casassg/corrent/tree/master/dags
>
> This approach is highly inspired from some alternatives that have lately
> appeared on ETL pipelines world (ex: Dagster) that support more functional
> ways of defining DAGs.
>
> I believe this approach could benefit the Airflow community to grow. What
> do you think? Any thoughts on functional DAGs? Would love to hear what are
> pros and cons of this expansion. To me, it seems highly linked to the
> Lineage discussion happening at the same time, as both discussions are
> around improving data pipelines.
>
> Thanks!
>
> Gerard Casas Saez
> Twitter | Cortex | @casassaez
>