You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Aritra Basu <ar...@gmail.com> on 2025/05/01 02:19:16 UTC
Re: [DISCUSS] Example dags
I'd be happy to help with the proposal as well Jens, you can count me in.
--
Regards,
Aritra Basu
On Thu, 1 May 2025, 2:43 am Jarek Potiuk, <ja...@potiuk.com> wrote:
> Very nice summary Jens and I 100% agree with the proposal.
>
> And BTW, this is not as difficult to implement as we might think. With
> bundle implementation, making providers contribute their own
> "example_bundle" seems like a very natural thing to do - this is actually
> what I implemented in order to show the "standard" example dags in Breeze.
> And extending provider discovery to also provide list of examples embedded
> in the provider should be super-easy as well, I am about to create a PR
> that moves the new queues to be "provided" by providers rather than
> embedded into "common.messaging", so I am happy to also do similar thing
> for examples and happy to join the 2-5 people team to work out the story
> told by examples. That would be really nice thing for the users/
>
> J
>
>
> On Wed, Apr 30, 2025 at 11:06 PM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
>
> > I retract my previous answer and agree with Jens. Well said! That's a
> > bigger lift to get it done, so maybe not part of the immediate migration,
> > but it would be a nice overall improvement project.
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Jens Scheffler <j_...@gmx.de.INVALID>
> > Sent: Wednesday, April 30, 2025 1:56 PM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXT] [DISCUSS] Example dags
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> > Thanks for opening the discussion.
> >
> > I have since a longer time the idea that we need to clean and re-work
> > the examples. I like how examples are used in some places in docs and
> > that examples. But at least 50% of them do not add real value to be
> > loaded. I think these are too many.
> >
> > I'd propose to take some effort and consolidate such that (1) a reduced
> > but meaningful number of examples is available which (2) are all used
> > and referenced for details and features in docs [=means the examples
> > should cover all core features of the product with examples] and (3)
> > maybe even tell a storyline. I remember from my former times all the
> > microsoft trainings I received with "northwind" database as example. It
> > would be cool having examples also around some artificial but realistic
> > use case where we can "eat your own dog food" implement some best
> > practices with proper documentation. (4) bonus really could be that if
> > you add a provider that each provider might bring another piece of the
> > puzzle to the overall use case [=e.g. once you add kafka provider one
> > example polls for customer survey events]
> >
> > I assume this would be a bit tooo much to have a creative discussion on
> > the devlist... but if there are 2-5 people who like to join, would there
> > be somebody else having a creative mind such that we can think about a
> > storyline and rework examples to this?
> >
> > (And the technical layer I also agree to option 1 of having bundles per
> > provider added also would be cool. Then also taggin all examples
> > consistently and adding tags per provider for easier filtering etc...
> > some kind of housekeeping and beautifying would be cool (I also once
> > proposed to add proper docs and display names to all examples to have
> > them pretty)
> >
> > Options 2+3 would not be needed if we make some cleanup and
> > consolidation and get the number down to a meaningful number
> >
> > On 30.04.25 22:06, Ferruzzi, Dennis wrote:
> > > I think I agree with option1, if the user chose to show example_dags
> > then they should get what they asked for. Option 2 feels like a cleaner
> > option, essentially curating the examples for new users, but it's
> opening a
> > can of worms debating which are "important" and where we draw the line,
> > etc. So I think I'd go with 1.
> > >
> > >
> > > - ferruzzi
> > >
> > >
> > > ________________________________
> > > From: Jarek Potiuk <ja...@potiuk.com>
> > > Sent: Wednesday, April 30, 2025 10:53 AM
> > > To: dev@airflow.apache.org
> > > Subject: RE: [EXT] [DISCUSS] Example dags
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si
> vous
> > ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain que le contenu ne présente aucun risque.
> > >
> > >
> > > The PR is finally green, sources updated, example dags moved to
> > "standard provider", yet in breeze they are visible. I run all the local
> > testing in Breeze for the moved dags and they work nicely.
> > >
> > > I think we might continue discussing what should be the "final"
> approach
> > - but for now - merging that one seems like a good idea. Unless someone
> > objects ?
> > >
> > > J.
> > >
> > >
> > > On Wed, Apr 30, 2025 at 5:16 PM Jarek Potiuk <jarek@potiuk.com<mailto:
> > jarek@potiuk.com>> wrote:
> > > Yep - all the dags will be visible in `breeze` in my PR
> > https://github.com/apache/airflow/pull/49978 - just pushed a change for
> > it:
> > >
> > > [image.png]
> > >
> > > On Wed, Apr 30, 2025 at 4:31 PM Jarek Potiuk <jarek@potiuk.com<mailto:
> > jarek@potiuk.com>> wrote:
> > > Answering Constance (somewhat it went to another thread:)
> > >
> > >> I know we rely on the example dags for dev purposes, especially for UI
> > > development. As part of moving them out of core, can we update breeze
> to
> > > automatically load them from their new home?
> > >
> > > Yes. Very good point. I strongly think we need to do it. And it's
> part
> > of the issue :).
> > >
> > > I already - in my PR - made it so in `breeze` and local venv where
> > `tests` are part of the system we have now two "example" bundles - the
> > second one coming from the standard provider. That was one of the reasons
> > my PR was not green initially - because also some of our tests in core
> > assume that "standard" provider examples are present. I adjusted some of
> > them to expect "simplest" dag - where it made no difference, but I also
> > created an extra dag bundle to add the standard provider in case the test
> > system is importable locally in tests - so far so good.
> > >
> > > But I think in order to make things **really** work for breeze
> > installation (and in local venv) - we would have to make it so the
> example
> > dags are:
> > >
> > > a) available in the provider packages
> > > b) airflow should discover them from there
> > >
> > > I can (quickly) make a hack for Breeze (that's the question of adding
> > standard provider's "tests" to PYTHONPATH) - in this PR - so It will
> cover
> > that one and we will be able to regenerate documentation with links to
> > sources.
> > >
> > > But I think in this discussion (and follow up PR) is to answer the
> > question of what users should see when they enable it in "installed"
> > airflow.
> > >
> > > * should users see the standard provider package dags when they
> install
> > airflow locally from packages and say "load_example_dags".
> > >
> > > if we think "yes", then we generally need to include example_dags in
> > provider packages (they are excluded now) - and we can do it, no problem.
> > > Now - I think we should also decide what users should "see" in this
> case
> > - my first reaction would be "standard" + "simplest" + "maybe a few
> others?
> > (which ones) ?"
> > >
> > > J.
> > >
> > >
> > > On Wed, Apr 30, 2025 at 1:12 PM Kaxil Naik <kaxilnaik@gmail.com
> <mailto:
> > kaxilnaik@gmail.com>> wrote:
> > > just an idea: With "git bundles", maybe example DAGs from standard
> > provider
> > > can be dynamically imported?
> > >
> > > But from your options: (1) seems reasonable
> > >
> > > On Wed, 30 Apr 2025 at 15:07, Jarek Potiuk <jarek@potiuk.com<mailto:
> > jarek@potiuk.com>> wrote:
> > >
> > >> Hello here.
> > >>
> > >> While fixing a bug in the doc generation to bring "source" link to
> > example
> > >> dags https://github.com/apache/airflow/pull/49978 I realized that we
> > have
> > >> not moved a number of python/bash etc. example dags to standard
> > provider -
> > >> where they now belong. I do it as part of my PR - but as result the
> > number
> > >> of "example_dags" shown in the UI when "load example dags" is true is
> > going
> > >> to decrease (and also they are not necessarily the dags you would
> really
> > >> want to see).
> > >>
> > >> I thought maybe about a follow up PR where we will do a few things:
> > >>
> > >> 1) allow to show example dags also from providers (or maybe just
> > standard
> > >> provider)
> > >> 2) limit a number of dags shown in airflow when "load example dags"
> > config
> > >> is set - for example by tags - to only see those that are really
> useful
> > (I
> > >> found the bash/python/comlex and few others really useful for quick
> > >> demo/testing - maybe there are others).
> > >> 3) We could optionally have "show all examples" setting for the
> "really
> > >> power user case" and make it easy to test things in airfow UI
> > >> out-of-the-box - or even limit it per provider.
> > >>
> > >> WDYT ?
> > >>
> > >> J
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Example dags
Posted by Aritra Basu <ar...@gmail.com>.
I was going to start a convo on it as well, wouldn't mind taking point on
this
--
Regards,
Aritra Basu
On Wed, 7 May 2025, 4:31 pm Jarek Potiuk, <ja...@potiuk.com> wrote:
> Indeed
>
> On Wed, May 7, 2025 at 12:35 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Worth adding it as an agenda item for dev call -- I am sure more folks
> > might want to help out on this one.
> >
> > On Wed, 7 May 2025 at 15:44, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I think it would be great to get someone to take a lead and define what
> > we
> > > should do here. From what I see (and slack discussion here
> > >
> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746587477484579
> > ),
> > > I
> > > think what we need now is to make a concrete proposal on
> > >
> > > * which examples should stay
> > > * should we include the standard ones
> > > * should we enable other example dags as option (and how)
> > >
> > > I don't feel competent on defining those, - as I mostly work on the CI
> > /DEV
> > > to support whatever is needed by those who need example dags for
> > something
> > > - but if someone who actually uses example dags in different scenarios
> > > could comment and come with proposals how we can treat them, i am happy
> > to
> > > support and even implement whatever - including the discovery of
> example
> > > DAGs from providers,
> > >
> > > J,
> > >
> > >
> > >
> > >
> > >
> > > On Thu, May 1, 2025 at 9:34 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > > Yes, probably too many example dags. Probably this is a
> consequence
> > of
> > > > using them as test dags.
> > > >
> > > > Just comment on that - I think it's more a consequence of not having
> a
> > > way
> > > > to separate those. The idea of using test dags for examples and code
> > > > snippets is an extremely sound one because it has this nice property
> > that
> > > > the examples are not going out-of-sync (which happens very quickly) -
> > but
> > > > yeah, what we are really discussing here is how to show such dags
> > > > selectively by-default n the UI without changing the dual-purpose of
> > > them.
> > > > And likely have a way to be able to show more or less of them if you
> > > > are the "dev" tester.
> > > >
> > > > J.
> > > >
> > > > On Thu, May 1, 2025 at 6:55 AM Daniel Standish
> > > > <da...@astronomer.io.invalid> wrote:
> > > >
> > > >> Yeah, along Jens lines...
> > > >>
> > > >> Yes, probably too many example dags. Probably this is a consequence
> > of
> > > >> using them as test dags.
> > > >>
> > > >> Probably ideally we are more selective in curating example dags and
> > > >> distinguishing them from test dags.
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
Indeed
On Wed, May 7, 2025 at 12:35 PM Kaxil Naik <ka...@gmail.com> wrote:
> Worth adding it as an agenda item for dev call -- I am sure more folks
> might want to help out on this one.
>
> On Wed, 7 May 2025 at 15:44, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I think it would be great to get someone to take a lead and define what
> we
> > should do here. From what I see (and slack discussion here
> > https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746587477484579
> ),
> > I
> > think what we need now is to make a concrete proposal on
> >
> > * which examples should stay
> > * should we include the standard ones
> > * should we enable other example dags as option (and how)
> >
> > I don't feel competent on defining those, - as I mostly work on the CI
> /DEV
> > to support whatever is needed by those who need example dags for
> something
> > - but if someone who actually uses example dags in different scenarios
> > could comment and come with proposals how we can treat them, i am happy
> to
> > support and even implement whatever - including the discovery of example
> > DAGs from providers,
> >
> > J,
> >
> >
> >
> >
> >
> > On Thu, May 1, 2025 at 9:34 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > Yes, probably too many example dags. Probably this is a consequence
> of
> > > using them as test dags.
> > >
> > > Just comment on that - I think it's more a consequence of not having a
> > way
> > > to separate those. The idea of using test dags for examples and code
> > > snippets is an extremely sound one because it has this nice property
> that
> > > the examples are not going out-of-sync (which happens very quickly) -
> but
> > > yeah, what we are really discussing here is how to show such dags
> > > selectively by-default n the UI without changing the dual-purpose of
> > them.
> > > And likely have a way to be able to show more or less of them if you
> > > are the "dev" tester.
> > >
> > > J.
> > >
> > > On Thu, May 1, 2025 at 6:55 AM Daniel Standish
> > > <da...@astronomer.io.invalid> wrote:
> > >
> > >> Yeah, along Jens lines...
> > >>
> > >> Yes, probably too many example dags. Probably this is a consequence
> of
> > >> using them as test dags.
> > >>
> > >> Probably ideally we are more selective in curating example dags and
> > >> distinguishing them from test dags.
> > >>
> > >
> >
>
Re: [DISCUSS] Example dags
Posted by Kaxil Naik <ka...@gmail.com>.
Worth adding it as an agenda item for dev call -- I am sure more folks
might want to help out on this one.
On Wed, 7 May 2025 at 15:44, Jarek Potiuk <ja...@potiuk.com> wrote:
> I think it would be great to get someone to take a lead and define what we
> should do here. From what I see (and slack discussion here
> https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746587477484579),
> I
> think what we need now is to make a concrete proposal on
>
> * which examples should stay
> * should we include the standard ones
> * should we enable other example dags as option (and how)
>
> I don't feel competent on defining those, - as I mostly work on the CI /DEV
> to support whatever is needed by those who need example dags for something
> - but if someone who actually uses example dags in different scenarios
> could comment and come with proposals how we can treat them, i am happy to
> support and even implement whatever - including the discovery of example
> DAGs from providers,
>
> J,
>
>
>
>
>
> On Thu, May 1, 2025 at 9:34 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > Yes, probably too many example dags. Probably this is a consequence of
> > using them as test dags.
> >
> > Just comment on that - I think it's more a consequence of not having a
> way
> > to separate those. The idea of using test dags for examples and code
> > snippets is an extremely sound one because it has this nice property that
> > the examples are not going out-of-sync (which happens very quickly) - but
> > yeah, what we are really discussing here is how to show such dags
> > selectively by-default n the UI without changing the dual-purpose of
> them.
> > And likely have a way to be able to show more or less of them if you
> > are the "dev" tester.
> >
> > J.
> >
> > On Thu, May 1, 2025 at 6:55 AM Daniel Standish
> > <da...@astronomer.io.invalid> wrote:
> >
> >> Yeah, along Jens lines...
> >>
> >> Yes, probably too many example dags. Probably this is a consequence of
> >> using them as test dags.
> >>
> >> Probably ideally we are more selective in curating example dags and
> >> distinguishing them from test dags.
> >>
> >
>
Re: [DISCUSS] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
I think it would be great to get someone to take a lead and define what we
should do here. From what I see (and slack discussion here
https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1746587477484579), I
think what we need now is to make a concrete proposal on
* which examples should stay
* should we include the standard ones
* should we enable other example dags as option (and how)
I don't feel competent on defining those, - as I mostly work on the CI /DEV
to support whatever is needed by those who need example dags for something
- but if someone who actually uses example dags in different scenarios
could comment and come with proposals how we can treat them, i am happy to
support and even implement whatever - including the discovery of example
DAGs from providers,
J,
On Thu, May 1, 2025 at 9:34 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > Yes, probably too many example dags. Probably this is a consequence of
> using them as test dags.
>
> Just comment on that - I think it's more a consequence of not having a way
> to separate those. The idea of using test dags for examples and code
> snippets is an extremely sound one because it has this nice property that
> the examples are not going out-of-sync (which happens very quickly) - but
> yeah, what we are really discussing here is how to show such dags
> selectively by-default n the UI without changing the dual-purpose of them.
> And likely have a way to be able to show more or less of them if you
> are the "dev" tester.
>
> J.
>
> On Thu, May 1, 2025 at 6:55 AM Daniel Standish
> <da...@astronomer.io.invalid> wrote:
>
>> Yeah, along Jens lines...
>>
>> Yes, probably too many example dags. Probably this is a consequence of
>> using them as test dags.
>>
>> Probably ideally we are more selective in curating example dags and
>> distinguishing them from test dags.
>>
>
Re: [DISCUSS] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
> Yes, probably too many example dags. Probably this is a consequence of
using them as test dags.
Just comment on that - I think it's more a consequence of not having a way
to separate those. The idea of using test dags for examples and code
snippets is an extremely sound one because it has this nice property that
the examples are not going out-of-sync (which happens very quickly) - but
yeah, what we are really discussing here is how to show such dags
selectively by-default n the UI without changing the dual-purpose of them.
And likely have a way to be able to show more or less of them if you
are the "dev" tester.
J.
On Thu, May 1, 2025 at 6:55 AM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> Yeah, along Jens lines...
>
> Yes, probably too many example dags. Probably this is a consequence of
> using them as test dags.
>
> Probably ideally we are more selective in curating example dags and
> distinguishing them from test dags.
>
Re: [VOTE] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool! I think this is something I learned - people look (if at all) at the
subjects.And if they don't - this means that they are not paying attention,
and it's fine not to take part in a vote when you had a chance and saw the
subject, but you chose not to pay attention :D
On Sun, Jun 22, 2025 at 11:17 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:
> Okay, then sorry for the noise... cancelling the VOTE and switching to a
> LAZY CONSENSUS in a fresh email thread.
>
> On 22.06.25 23:00, Jarek Potiuk wrote:
> > I think it would be better to call for LAZY CONSENSUS with `[LAZY
> > CONSENSUS]` title - calling a vote is indeed not needed, but having a
> vote
> > in a '[DISCUSS]` thread might be misleadin - one of the reason to have
> > `[PREFIX]' is to make them aware that there is a vote or consensus going
> > on, where they might have to take look - where '[DISCUSS]' is not really
> > binding for decision.
> >
> > On Sun, Jun 22, 2025 at 10:50 PM Jens Scheffler
> <j_...@gmx.de.invalid>
> > wrote:
> >
> >> Hi,
> >>
> >> as the DICSUSS emails silcenced and also the number of ideas/comments
> >> flattened I wanted to call for a VOTE for the example dag
> >> cleanup/refurbish.
> >>
> >> Together with Arita, Shubham, Desai and Jarek we have consolidated
> >> idea/plans/tasks in
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
> >> to be the base of implementation.
> >>
> >> I believe an "formal/official" vote is not needed, could also be just a
> >> set of PRs being merged but as it is a larger storyline change I'd cast
> >> the vote to align plans. Comments on the docs and small extensions are
> >> still welcome otherwise target would be to implement the consolidation
> >> and storyline as documented.
> >>
> >> The vote will run for ~5 days and last till next Friday, June 27th 23:59
> >> CEST.
> >>
> >> Everyone is encouraged to vote, although only PMC members and
> >> Committer's votes are considered binding.
> >>
> >> consider my +1 (binding)
> >>
> >> Jens
> >>
> >> On 11.05.25 16:24, Jens Scheffler wrote:
> >>> Hi,
> >>>
> >>> following-up on the discussion we had and the verbal discussion in
> >>> Airflow 3 Dev call May 22nd (see notes in
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025
> )
> >>
> >>> I propose the following next steps:
> >>>
> >>> 1) start a separate interactive discussion in Slack in channel
> >>> #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466)
> >>> - Note: If you have the feeling we need to discuss in Devlist, please
> >>> also feel free to replay to the thread here. Butmaybe we need to have
> >>> a bit more interaction w/o generating too much noise in the devlist)
> >>>
> >>> 2) Within the discussion document some targets and tasks as a concept.
> >>> I started a skeleton in
> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
> >>> - Feel free to extend, comment!
> >>>
> >>> 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
> >>> propose a follow-up [VOTE] in the devlist validating agreement
> >>>
> >>> Everybody who likes to join and engage, please feel free to join
> >>> #sig-examples in Slack or let me know!
> >>>
> >>> Jens
> >>>
> >>> On 01.05.25 06:54, Daniel Standish wrote:
> >>>> Yeah, along Jens lines...
> >>>>
> >>>> Yes, probably too many example dags. Probably this is a consequence
> of
> >>>> using them as test dags.
> >>>>
> >>>> Probably ideally we are more selective in curating example dags and
> >>>> distinguishing them from test dags.
> >>>>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [VOTE] Example dags
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Okay, then sorry for the noise... cancelling the VOTE and switching to a
LAZY CONSENSUS in a fresh email thread.
On 22.06.25 23:00, Jarek Potiuk wrote:
> I think it would be better to call for LAZY CONSENSUS with `[LAZY
> CONSENSUS]` title - calling a vote is indeed not needed, but having a vote
> in a '[DISCUSS]` thread might be misleadin - one of the reason to have
> `[PREFIX]' is to make them aware that there is a vote or consensus going
> on, where they might have to take look - where '[DISCUSS]' is not really
> binding for decision.
>
> On Sun, Jun 22, 2025 at 10:50 PM Jens Scheffler <j_...@gmx.de.invalid>
> wrote:
>
>> Hi,
>>
>> as the DICSUSS emails silcenced and also the number of ideas/comments
>> flattened I wanted to call for a VOTE for the example dag
>> cleanup/refurbish.
>>
>> Together with Arita, Shubham, Desai and Jarek we have consolidated
>> idea/plans/tasks in
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
>> to be the base of implementation.
>>
>> I believe an "formal/official" vote is not needed, could also be just a
>> set of PRs being merged but as it is a larger storyline change I'd cast
>> the vote to align plans. Comments on the docs and small extensions are
>> still welcome otherwise target would be to implement the consolidation
>> and storyline as documented.
>>
>> The vote will run for ~5 days and last till next Friday, June 27th 23:59
>> CEST.
>>
>> Everyone is encouraged to vote, although only PMC members and
>> Committer's votes are considered binding.
>>
>> consider my +1 (binding)
>>
>> Jens
>>
>> On 11.05.25 16:24, Jens Scheffler wrote:
>>> Hi,
>>>
>>> following-up on the discussion we had and the verbal discussion in
>>> Airflow 3 Dev call May 22nd (see notes in
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025)
>>
>>> I propose the following next steps:
>>>
>>> 1) start a separate interactive discussion in Slack in channel
>>> #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466)
>>> - Note: If you have the feeling we need to discuss in Devlist, please
>>> also feel free to replay to the thread here. Butmaybe we need to have
>>> a bit more interaction w/o generating too much noise in the devlist)
>>>
>>> 2) Within the discussion document some targets and tasks as a concept.
>>> I started a skeleton in
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
>>> - Feel free to extend, comment!
>>>
>>> 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
>>> propose a follow-up [VOTE] in the devlist validating agreement
>>>
>>> Everybody who likes to join and engage, please feel free to join
>>> #sig-examples in Slack or let me know!
>>>
>>> Jens
>>>
>>> On 01.05.25 06:54, Daniel Standish wrote:
>>>> Yeah, along Jens lines...
>>>>
>>>> Yes, probably too many example dags. Probably this is a consequence of
>>>> using them as test dags.
>>>>
>>>> Probably ideally we are more selective in curating example dags and
>>>> distinguishing them from test dags.
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [VOTE] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
I think it would be better to call for LAZY CONSENSUS with `[LAZY
CONSENSUS]` title - calling a vote is indeed not needed, but having a vote
in a '[DISCUSS]` thread might be misleadin - one of the reason to have
`[PREFIX]' is to make them aware that there is a vote or consensus going
on, where they might have to take look - where '[DISCUSS]' is not really
binding for decision.
On Sun, Jun 22, 2025 at 10:50 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:
> Hi,
>
> as the DICSUSS emails silcenced and also the number of ideas/comments
> flattened I wanted to call for a VOTE for the example dag
> cleanup/refurbish.
>
> Together with Arita, Shubham, Desai and Jarek we have consolidated
> idea/plans/tasks in
> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
> to be the base of implementation.
>
> I believe an "formal/official" vote is not needed, could also be just a
> set of PRs being merged but as it is a larger storyline change I'd cast
> the vote to align plans. Comments on the docs and small extensions are
> still welcome otherwise target would be to implement the consolidation
> and storyline as documented.
>
> The vote will run for ~5 days and last till next Friday, June 27th 23:59
> CEST.
>
> Everyone is encouraged to vote, although only PMC members and
> Committer's votes are considered binding.
>
> consider my +1 (binding)
>
> Jens
>
> On 11.05.25 16:24, Jens Scheffler wrote:
> > Hi,
> >
> > following-up on the discussion we had and the verbal discussion in
> > Airflow 3 Dev call May 22nd (see notes in
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025)
>
> > I propose the following next steps:
> >
> > 1) start a separate interactive discussion in Slack in channel
> > #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466)
> > - Note: If you have the feeling we need to discuss in Devlist, please
> > also feel free to replay to the thread here. Butmaybe we need to have
> > a bit more interaction w/o generating too much noise in the devlist)
> >
> > 2) Within the discussion document some targets and tasks as a concept.
> > I started a skeleton in
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
> > - Feel free to extend, comment!
> >
> > 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
> > propose a follow-up [VOTE] in the devlist validating agreement
> >
> > Everybody who likes to join and engage, please feel free to join
> > #sig-examples in Slack or let me know!
> >
> > Jens
> >
> > On 01.05.25 06:54, Daniel Standish wrote:
> >> Yeah, along Jens lines...
> >>
> >> Yes, probably too many example dags. Probably this is a consequence of
> >> using them as test dags.
> >>
> >> Probably ideally we are more selective in curating example dags and
> >> distinguishing them from test dags.
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
[VOTE] Example dags
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Hi,
as the DICSUSS emails silcenced and also the number of ideas/comments
flattened I wanted to call for a VOTE for the example dag cleanup/refurbish.
Together with Arita, Shubham, Desai and Jarek we have consolidated
idea/plans/tasks in
https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
to be the base of implementation.
I believe an "formal/official" vote is not needed, could also be just a
set of PRs being merged but as it is a larger storyline change I'd cast
the vote to align plans. Comments on the docs and small extensions are
still welcome otherwise target would be to implement the consolidation
and storyline as documented.
The vote will run for ~5 days and last till next Friday, June 27th 23:59
CEST.
Everyone is encouraged to vote, although only PMC members and
Committer's votes are considered binding.
consider my +1 (binding)
Jens
On 11.05.25 16:24, Jens Scheffler wrote:
> Hi,
>
> following-up on the discussion we had and the verbal discussion in
> Airflow 3 Dev call May 22nd (see notes in
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025)
> I propose the following next steps:
>
> 1) start a separate interactive discussion in Slack in channel
> #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466)
> - Note: If you have the feeling we need to discuss in Devlist, please
> also feel free to replay to the thread here. Butmaybe we need to have
> a bit more interaction w/o generating too much noise in the devlist)
>
> 2) Within the discussion document some targets and tasks as a concept.
> I started a skeleton in
> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
> - Feel free to extend, comment!
>
> 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
> propose a follow-up [VOTE] in the devlist validating agreement
>
> Everybody who likes to join and engage, please feel free to join
> #sig-examples in Slack or let me know!
>
> Jens
>
> On 01.05.25 06:54, Daniel Standish wrote:
>> Yeah, along Jens lines...
>>
>> Yes, probably too many example dags. Probably this is a consequence of
>> using them as test dags.
>>
>> Probably ideally we are more selective in curating example dags and
>> distinguishing them from test dags.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Example dags
Posted by Amogh Desai <am...@gmail.com>.
Nice!
Thanks & Regards,
Amogh Desai
On Sat, May 31, 2025 at 4:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> And another small follow-up before we release 3.0.2 as Jed pointed out in
> the previous PR, those example dags from standard provider do not really
> fit the same pattern as system tests + example dags in google, amazon and
> other providers - so we should likely put them in example_dags folder and
> for sure remove test_common snippets from them before releasing 3.0.2.
> https://github.com/apache/airflow/pull/51260
>
> On Thu, May 29, 2025 at 7:55 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > As discussed at the last dev call - before we release 3.0.2 , I brought
> > back example_dags back to "airflow" package - until we do the complete
> > "Example Refurbish" - this one should bring back the same "experience"
> for
> > example dags as before - including standalone airflow and airflow
> installed
> > in venv:
> >
> > https://github.com/apache/airflow/pull/51192
> >
> > This is done with a bit of cheating. The "standard" examples are still in
> > "standard" provider - thanks to that all the docs work nicely, but in our
> > repo I created a symbolic link "standard" in the 'examples" directory to
> > the "standard" examples - and I removed all the "hacks" to read them from
> > standard provider in breeze. I checked that it nicely works for package
> > preparation (the standard symbolic links are copied as folder to both
> sdist
> > and whl. packages)
> >
> > This way "source of truth" - standard examples are in the standard
> > provider, but they are also present in the main airflow folder. This has
> a
> > bit of a watch-out, because when we cherry-pick it to 3-0-test, we should
> > also probably cherry-pick any changes to standard provider's examples -
> for
> > now https://github.com/apache/airflow/pull/50864 (and it's predecessor)
> > is the only change v3-0-test vs. main.
> >
> >
> > J.
> >
> >
> > On Mon, May 12, 2025 at 8:44 AM Amogh Desai <am...@gmail.com>
> > wrote:
> >
> >> Sounds great Jens, thanks for creating the channel and
> >> thanks for updating us on it.
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >> On Sun, May 11, 2025 at 7:54 PM Jens Scheffler
> <j_scheffler@gmx.de.invalid
> >> >
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > following-up on the discussion we had and the verbal discussion in
> >> > Airflow 3 Dev call May 22nd (see notes in
> >> >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025
> >> )
> >> >
> >> > I propose the following next steps:
> >> >
> >> > 1) start a separate interactive discussion in Slack in channel
> >> > #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466)
> -
> >> > Note: If you have the feeling we need to discuss in Devlist, please
> also
> >> > feel free to replay to the thread here. Butmaybe we need to have a bit
> >> > more interaction w/o generating too much noise in the devlist)
> >> >
> >> > 2) Within the discussion document some targets and tasks as a
> concept. I
> >> > started a skeleton in
> >> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
> >> -
> >> > Feel free to extend, comment!
> >> >
> >> > 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
> >> > propose a follow-up [VOTE] in the devlist validating agreement
> >> >
> >> > Everybody who likes to join and engage, please feel free to join
> >> > #sig-examples in Slack or let me know!
> >> >
> >> > Jens
> >> >
> >> > On 01.05.25 06:54, Daniel Standish wrote:
> >> > > Yeah, along Jens lines...
> >> > >
> >> > > Yes, probably too many example dags. Probably this is a consequence
> >> of
> >> > > using them as test dags.
> >> > >
> >> > > Probably ideally we are more selective in curating example dags and
> >> > > distinguishing them from test dags.
> >> > >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > For additional commands, e-mail: dev-help@airflow.apache.org
> >> >
> >> >
> >>
> >
>
Re: [DISCUSS] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
And another small follow-up before we release 3.0.2 as Jed pointed out in
the previous PR, those example dags from standard provider do not really
fit the same pattern as system tests + example dags in google, amazon and
other providers - so we should likely put them in example_dags folder and
for sure remove test_common snippets from them before releasing 3.0.2.
https://github.com/apache/airflow/pull/51260
On Thu, May 29, 2025 at 7:55 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> As discussed at the last dev call - before we release 3.0.2 , I brought
> back example_dags back to "airflow" package - until we do the complete
> "Example Refurbish" - this one should bring back the same "experience" for
> example dags as before - including standalone airflow and airflow installed
> in venv:
>
> https://github.com/apache/airflow/pull/51192
>
> This is done with a bit of cheating. The "standard" examples are still in
> "standard" provider - thanks to that all the docs work nicely, but in our
> repo I created a symbolic link "standard" in the 'examples" directory to
> the "standard" examples - and I removed all the "hacks" to read them from
> standard provider in breeze. I checked that it nicely works for package
> preparation (the standard symbolic links are copied as folder to both sdist
> and whl. packages)
>
> This way "source of truth" - standard examples are in the standard
> provider, but they are also present in the main airflow folder. This has a
> bit of a watch-out, because when we cherry-pick it to 3-0-test, we should
> also probably cherry-pick any changes to standard provider's examples - for
> now https://github.com/apache/airflow/pull/50864 (and it's predecessor)
> is the only change v3-0-test vs. main.
>
>
> J.
>
>
> On Mon, May 12, 2025 at 8:44 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> Sounds great Jens, thanks for creating the channel and
>> thanks for updating us on it.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Sun, May 11, 2025 at 7:54 PM Jens Scheffler <j_scheffler@gmx.de.invalid
>> >
>> wrote:
>>
>> > Hi,
>> >
>> > following-up on the discussion we had and the verbal discussion in
>> > Airflow 3 Dev call May 22nd (see notes in
>> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025
>> )
>> >
>> > I propose the following next steps:
>> >
>> > 1) start a separate interactive discussion in Slack in channel
>> > #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466) -
>> > Note: If you have the feeling we need to discuss in Devlist, please also
>> > feel free to replay to the thread here. Butmaybe we need to have a bit
>> > more interaction w/o generating too much noise in the devlist)
>> >
>> > 2) Within the discussion document some targets and tasks as a concept. I
>> > started a skeleton in
>> > https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish
>> -
>> > Feel free to extend, comment!
>> >
>> > 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
>> > propose a follow-up [VOTE] in the devlist validating agreement
>> >
>> > Everybody who likes to join and engage, please feel free to join
>> > #sig-examples in Slack or let me know!
>> >
>> > Jens
>> >
>> > On 01.05.25 06:54, Daniel Standish wrote:
>> > > Yeah, along Jens lines...
>> > >
>> > > Yes, probably too many example dags. Probably this is a consequence
>> of
>> > > using them as test dags.
>> > >
>> > > Probably ideally we are more selective in curating example dags and
>> > > distinguishing them from test dags.
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>
Re: [DISCUSS] Example dags
Posted by Jarek Potiuk <ja...@potiuk.com>.
As discussed at the last dev call - before we release 3.0.2 , I brought
back example_dags back to "airflow" package - until we do the complete
"Example Refurbish" - this one should bring back the same "experience" for
example dags as before - including standalone airflow and airflow installed
in venv:
https://github.com/apache/airflow/pull/51192
This is done with a bit of cheating. The "standard" examples are still in
"standard" provider - thanks to that all the docs work nicely, but in our
repo I created a symbolic link "standard" in the 'examples" directory to
the "standard" examples - and I removed all the "hacks" to read them from
standard provider in breeze. I checked that it nicely works for package
preparation (the standard symbolic links are copied as folder to both sdist
and whl. packages)
This way "source of truth" - standard examples are in the standard
provider, but they are also present in the main airflow folder. This has a
bit of a watch-out, because when we cherry-pick it to 3-0-test, we should
also probably cherry-pick any changes to standard provider's examples - for
now https://github.com/apache/airflow/pull/50864 (and it's predecessor) is
the only change v3-0-test vs. main.
J.
On Mon, May 12, 2025 at 8:44 AM Amogh Desai <am...@gmail.com>
wrote:
> Sounds great Jens, thanks for creating the channel and
> thanks for updating us on it.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Sun, May 11, 2025 at 7:54 PM Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> wrote:
>
> > Hi,
> >
> > following-up on the discussion we had and the verbal discussion in
> > Airflow 3 Dev call May 22nd (see notes in
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025
> )
> >
> > I propose the following next steps:
> >
> > 1) start a separate interactive discussion in Slack in channel
> > #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466) -
> > Note: If you have the feeling we need to discuss in Devlist, please also
> > feel free to replay to the thread here. Butmaybe we need to have a bit
> > more interaction w/o generating too much noise in the devlist)
> >
> > 2) Within the discussion document some targets and tasks as a concept. I
> > started a skeleton in
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish -
> > Feel free to extend, comment!
> >
> > 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
> > propose a follow-up [VOTE] in the devlist validating agreement
> >
> > Everybody who likes to join and engage, please feel free to join
> > #sig-examples in Slack or let me know!
> >
> > Jens
> >
> > On 01.05.25 06:54, Daniel Standish wrote:
> > > Yeah, along Jens lines...
> > >
> > > Yes, probably too many example dags. Probably this is a consequence of
> > > using them as test dags.
> > >
> > > Probably ideally we are more selective in curating example dags and
> > > distinguishing them from test dags.
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Example dags
Posted by Amogh Desai <am...@gmail.com>.
Sounds great Jens, thanks for creating the channel and
thanks for updating us on it.
Thanks & Regards,
Amogh Desai
On Sun, May 11, 2025 at 7:54 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:
> Hi,
>
> following-up on the discussion we had and the verbal discussion in
> Airflow 3 Dev call May 22nd (see notes in
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025)
>
> I propose the following next steps:
>
> 1) start a separate interactive discussion in Slack in channel
> #sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466) -
> Note: If you have the feeling we need to discuss in Devlist, please also
> feel free to replay to the thread here. Butmaybe we need to have a bit
> more interaction w/o generating too much noise in the devlist)
>
> 2) Within the discussion document some targets and tasks as a concept. I
> started a skeleton in
> https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish -
> Feel free to extend, comment!
>
> 3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
> propose a follow-up [VOTE] in the devlist validating agreement
>
> Everybody who likes to join and engage, please feel free to join
> #sig-examples in Slack or let me know!
>
> Jens
>
> On 01.05.25 06:54, Daniel Standish wrote:
> > Yeah, along Jens lines...
> >
> > Yes, probably too many example dags. Probably this is a consequence of
> > using them as test dags.
> >
> > Probably ideally we are more selective in curating example dags and
> > distinguishing them from test dags.
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Example dags
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Hi,
following-up on the discussion we had and the verbal discussion in
Airflow 3 Dev call May 22nd (see notes in
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-22May2025)
I propose the following next steps:
1) start a separate interactive discussion in Slack in channel
#sig-examples (https://apache-airflow.slack.com/archives/C08RUFEL466) -
Note: If you have the feeling we need to discuss in Devlist, please also
feel free to replay to the thread here. Butmaybe we need to have a bit
more interaction w/o generating too much noise in the devlist)
2) Within the discussion document some targets and tasks as a concept. I
started a skeleton in
https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish -
Feel free to extend, comment!
3) Once our discussion in #sig-examples/Slack/Devlist is sttled I
propose a follow-up [VOTE] in the devlist validating agreement
Everybody who likes to join and engage, please feel free to join
#sig-examples in Slack or let me know!
Jens
On 01.05.25 06:54, Daniel Standish wrote:
> Yeah, along Jens lines...
>
> Yes, probably too many example dags. Probably this is a consequence of
> using them as test dags.
>
> Probably ideally we are more selective in curating example dags and
> distinguishing them from test dags.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Example dags
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
Yeah, along Jens lines...
Yes, probably too many example dags. Probably this is a consequence of
using them as test dags.
Probably ideally we are more selective in curating example dags and
distinguishing them from test dags.