You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Wei Lee <we...@apache.org> on 2025/02/17 07:27:17 UTC
Re: Airflow should deprecate the term "DAG" for end users
It seems that our current conclusion is to use "dag" or "Dag" instead of "DAG" whenever possible. Should we replace all "DAG" in the codebase with "dag" or "Dag"? If it's too late for that (which it might be đ€), should we at least avoid introducing new "DAG" in the following PRs?
Best,
Wei
On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> Here's a PR to use "dag" as a word in the new 3.0 UI:
> https://github.com/apache/airflow/pull/43325
> Let me know if that's the direction we want to go.
>
> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> <rb...@amazon.com.invalid> wrote:
>
> > I think Brent & Daniel summarized it best, "dag" is synonymous with
> > workflows in Airflow through the way we talk and explain what Airflow is
> > all about. Although folks would ask, I donât ever use the mathematical
> > definition of DAG.
> > It will be challenging and possibly confusing for many users making such a
> > change - I would rather direct that energy to appending the Oxford
> > definition of "dag" to include a reference to workflows in Airflow. __
> >
> > -- Rajesh
> >
> >
> >
> >
> >
> >
> > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > jarek@potiuk.com>> wrote:
> >
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and know
> > the content is safe.
> >
> >
> >
> >
> >
> >
> > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur externe.
> > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne pouvez
> > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain que
> > le contenu ne présente aucun risque.
> >
> >
> >
> >
> >
> >
> > From Guido's post:
> >
> >
> > "Around this time the renaming seems to have been renamed".
> >
> >
> > Naming is hard.
> >
> >
> > J.
> >
> >
> >
> >
> > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev510e@gmail.com <mailto:
> > droiddev510e@gmail.com>> wrote:
> >
> >
> > > Yes, completely agree with above comments, dag is an Airflow term now
> > > rather
> > > than just a "directed acyclic graph". Believe it or not, I've worked with
> > > work
> > > folks who've been using dags in Airflow for years now and are pro devs
> > but
> > > have trouble remembering the full form of a DAG! For them, dag = Airflow.
> > >
> > > workflow, pipeline, flow are surely better terms but will be a major
> > > behavior
> > > change for the developer community, so we'll need a solid plan on how to
> > > introduce it, when we do.
> > >
> > > While we discuss this, I'd like to share about the Great Renaming in core
> > > Python started by Guido back in 2009. We could probably get some
> > learnings
> > > from there? Who knows!
> > >
> > > Guido's blog (2009):
> > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > <https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html>
> > > Follow-up discussion (2024):
> > > https://discuss.python.org/t/finishing-the-great-renaming/54082 <
> > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > >
> > > Cheers,
> > > Omkar
> > >
> > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com <mailto:
> > weilee.rx@gmail.com>> wrote:
> > >
> > > > I think we should probably just accept it as an airflow term. At least,
> > > > thatâs how I understood it when I first used Airflow. I feel renaming
> > it
> > > at
> > > > this stage would require considerable effort from maintainers and
> > > existing
> > > > users without providing equivalent benefits.
> > > >
> > > > Best,
> > > > Wei
> > > >
> > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxilnaik@gmail.com
> > <ma...@gmail.com>> wrote:
> > > > >
> > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this discussion
> > > for
> > > > > Airflow 4 :) -- but right now it will cause too much disruption
> > > > >
> > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > <constance@astronomer.io.inva <ma...@astronomer.io.inva>lid>
> > wrote:
> > > > >
> > > > >> In my experience, when you ask those with Airflow experience what a
> > > dag
> > > > is,
> > > > >> they'll start talking about workflow attributes - stuff like dags
> > > being
> > > > a
> > > > >> series of steps or tasks with owners. The structure doesn't come up.
> > > > >>
> > > > >> Echo-ing others, at this point, my vote is to embrace the name and
> > > > >> de-emphasize the mathematical structure aspect.
> > > > >>
> > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva>lid>
> > > > >> wrote:
> > > > >>
> > > > >>> It's an interesting discussion and I remember struggling with this
> > > > when I
> > > > >>> started working with Airflow.
> > > > >>> But, I also agree with the viewpoint of it being an established
> > > concept
> > > > >> now
> > > > >>> regardless of the origin.
> > > > >>>
> > > > >>> I am personally leaning towards the perspective best expressed by
> > > > Daniel
> > > > >>> Standish and Brent of using Dag as a word, rather than the computer
> > > > >> science
> > > > >>> concept.
> > > > >>>
> > > > >>> Best regards,
> > > > >>> Vikram
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > >> <onikolas@amazon.com.inva <ma...@amazon.com.inva>lid
> > > > >>>>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> I agree with the general sentiment of: You're right Ryan, DAG
> > isn't
> > > > >> great
> > > > >>>> and I'd rather workflow, but changing it will cause much more
> > > wreckage
> > > > >>> than
> > > > >>>> it solves.
> > > > >>>>
> > > > >>>> Also agree with the idea to just move away from defining DAG. I
> > > think
> > > > >>>> we've been naturally doing that as a community for a while now
> > > anyway,
> > > > >> so
> > > > >>>> that feels like a natural step.
> > > > >>>>
> > > > >>>> Cheers,
> > > > >>>> Niko
> > > > >>>>
> > > > >>>> ________________________________
> > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>>
> > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > >>>> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end
> > > > >> users
> > > > >>>>
> > > > >>>> CAUTION: This email originated from outside of the organization.
> > Do
> > > > not
> > > > >>>> click links or open attachments unless you can confirm the sender
> > > and
> > > > >>> know
> > > > >>>> the content is safe.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
> > > > >> externe.
> > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous
> > ne
> > > > >>> pouvez
> > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> > > certain
> > > > >>> que
> > > > >>>> le contenu ne présente aucun risque.
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> Best argument in favour of keeping âdagsâ as a term â getting to
> > > > re-use
> > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > https://i.imgflip.com/1xhtwh.jpg>
> > > > >>>>
> > > > >>>> In all seriousness: I donât mind either way, both sides have good
> > > > >> reasons
> > > > >>>> presented.
> > > > >>>>
> > > > >>>> -a
> > > > >>>>
> > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > >>>>>
> > > > >>>>> Yeah just say, when asked where the name comes from, "well, no
> > one
> > > > >>>> actually
> > > > >>>>> knows but..." and then make something up.
> > > > >>>>>
> > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com>>
> > > > >> wrote:
> > > > >>>>>
> > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > tongue-twister,
> > > > >>>>>>
> > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <jarek@potiuk.com
> > <ma...@potiuk.com>>
> > > > >>> wrote:
> > > > >>>>>>
> > > > >>>>>>> I like what both Daniel and Brent wrote. I would very much want
> > > to
> > > > >> be
> > > > >>>>>> able
> > > > >>>>>>> to say just "dag" without explaining it further.
> > > > >>>>>>>
> > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > tongue-twister,
> > > > >>> and
> > > > >>>> I
> > > > >>>>>>> almost stutter on trying to recall how to pronounce it
> > properly.
> > > > >>>>>>>
> > > > >>>>>>> J.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > >>>>>> <brent@astronomer.io.inva <ma...@astronomer.io.inva>lid>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> I remember we explored renaming "DAG" when starting on AIP-38
> > to
> > > > >>>>>> modernize
> > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more descriptive of
> > > what
> > > > >>> one
> > > > >>>>>> is
> > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > implementation
> > > > >>>> detail.
> > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG" has
> > become
> > > > >>> "dag"
> > > > >>>>>> , a
> > > > >>>>>>>> word in its own right.
> > > > >>>>>>>>
> > > > >>>>>>>> Examples for "dag" are abound in community discussion, Airflow
> > > > >>> Summit
> > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace "dag".
> > A
> > > > >> user
> > > > >>>>>> just
> > > > >>>>>>>> needs to learn one new word vs the technical concept behind
> > that
> > > > >>>> word. I
> > > > >>>>>>>> think that is much less effort than refactoring so much code,
> > > > >>>>>>>> documentation, blog posts, stack overflow questions, etc.
> > > > >>>>>>>>
> > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain for
> > > > >>>> questionable
> > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I agree the
> > > > >>>>>>>> association
> > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > >>>>>>>>>
> > > > >>>>>>>>> And along those lines, I offer here some less invasive
> > > > >> mitigations.
> > > > >>>>>>>>>
> > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize the
> > math
> > > > >> nerd
> > > > >>>>>>>> origins
> > > > >>>>>>>>> of the name. That is to say, in docs / website / etc, *never
> > > > >>> define*
> > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph. Always
> > > > >> define
> > > > >>>> it
> > > > >>>>>>>> as a
> > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > >>>>>>>>>
> > > > >>>>>>>>> The "directed acyclic graph" part of it is like a historical
> > > > >>>> footnote,
> > > > >>>>>>>> and
> > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > >>>>>>>>>
> > > > >>>>>>>>> We could also start using lowercase in the docs in general
> > e.g.
> > > > >>>>>> writing
> > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc. The
> > > upper
> > > > >>> case
> > > > >>>>>>>> part
> > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in airlfow is
> > > just
> > > > >>> an
> > > > >>>>>>>>> airflow concept and the association with "DAGs" is not really
> > > > >>>>>> unhelpful.
> > > > >>>>>>>>>
> > > > >>>>>>>>> In other words embrace that "dag" in airflow is its own
> > thing,
> > > is
> > > > >>>>>>>>> *not* strictly
> > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows about
> > > > >>> anyway),
> > > > >>>>>> and
> > > > >>>>>>>>> tell them what it is in simple terms that normal people
> > > > >> understand.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > jarek@potiuk.com <ma...@potiuk.com>
> > > >
> > > > >>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> DAG is so embedded into what we do that it will be extremely
> > > > >>>>>>>> difficult to
> > > > >>>>>>>>>> get rid of it completely. Also I think it will make a lot of
> > > > >>>>>> "google"
> > > > >>>>>>>>>> searches and "stack overflow" searches not finding the right
> > > > >>>>>> answers.
> > > > >>>>>>>>> This
> > > > >>>>>>>>>> is one of the strengths of Airflow - besides the community
> > and
> > > > >>> ideas
> > > > >>>>>>>> that
> > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples, problems
> > and
> > > > >>>>>>>> solutions
> > > > >>>>>>>>>> you can so easily find (and we have to remember that all the
> > > AI
> > > > >>>>>>>> trained
> > > > >>>>>>>>> on
> > > > >>>>>>>>>> past data will be also rather poorly matching queries of
> > > people.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch. And if
> > we
> > > > >> do
> > > > >>> -
> > > > >>>>>> I
> > > > >>>>>>>>>> would be for using workflow or pipeline instead of `dag` if
> > > not
> > > > >>> the
> > > > >>>>>>>> above
> > > > >>>>>>>>>> reason, but I think I am here with Igor that it might cause
> > > more
> > > > >>>>>>>> problems
> > > > >>>>>>>>>> than it solves.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> But I am not 100% against - if others will think it's a good
> > > > >>> idea, I
> > > > >>>>>>>> am
> > > > >>>>>>>>> ok
> > > > >>>>>>>>>> with it.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> J,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to someone
> > new
> > > > >> to
> > > > >>>>>>>>> workflow
> > > > >>>>>>>>>>> orchestration. But it does also show the nature of being
> > > > >> acyclic.
> > > > >>>>>>>> Sure,
> > > > >>>>>>>>>> as
> > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it. Still, in
> > my
> > > > >>>>>>>> opinion,
> > > > >>>>>>>>>> there
> > > > >>>>>>>>>>> is generally no need for cyclic behavior in workflow
> > > > >>>>>> orchestration.
> > > > >>>>>>>>> Most
> > > > >>>>>>>>>>> (*if
> > > > >>>>>>>>>>> not all*) cases can be in some way can be covered using an
> > > > >>> acyclic
> > > > >>>>>>>>> manner
> > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I would want
> > > the
> > > > >>>>>>>>> "acyclic"
> > > > >>>>>>>>>>> word to stick.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Regards,
> > > > >>>>>>>>>>> Avi
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow Fan; so
> > many
> > > > >> new
> > > > >>>>>>>>> ideas.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced by the
> > > more
> > > > >>>>>>>> general
> > > > >>>>>>>>>>> Term
> > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri) Net
> > (AFN)
> > > > >>>>>> (maybe
> > > > >>>>>>>>>>> without
> > > > >>>>>>>>>>>> a direction);
> > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph Database.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG) might be
> > > > >>>>>>>> assigned
> > > > >>>>>>>>> to
> > > > >>>>>>>>>> an
> > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a library.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> A running Graph might have a different structure than a
> > > > >>>>>>>> configuration
> > > > >>>>>>>>>>>> Graph.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best Regards
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Bernd Ströhle
> > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > bernd.stroehle@gmail.com>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> -----Original Message-----
> > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA <mailto:
> > ikholopov@google.com.INVA>LID>
> > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org
> > >
> > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term "DAG" for
> > end
> > > > >>>>>> users
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, it is
> > part
> > > > >> of
> > > > >>>>>>>>> Airflow
> > > > >>>>>>>>>>>> DAG definition interface at so many levels, that any
> > attempt
> > > > >> to
> > > > >>>>>>>>> change
> > > > >>>>>>>>>> it
> > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The only
> > > thing
> > > > >>>>>>>> that is
> > > > >>>>>>>>>>> worse
> > > > >>>>>>>>>>>> than a poorly chosen name in the code is when there are
> > two
> > > > >> ways
> > > > >>>>>>>> to
> > > > >>>>>>>>>>> define
> > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials will
> > > suddenly
> > > > >>>>>>>> become
> > > > >>>>>>>>>>>> confusing as they all refer to workflows as "DAG"s.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> We are already at risk of scaring the users away with a
> > > number
> > > > >>>>>> of
> > > > >>>>>>>>>>> breaking
> > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking changes
> > > for
> > > > >>>>>> the
> > > > >>>>>>>>> most
> > > > >>>>>>>>>>>> basic things is not something that people are looking for.
> > > > >>>>>>>> Attempting
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>> change the fundamental terms will be interpreted as an
> > even
> > > > >>>>>>>> stronger
> > > > >>>>>>>>>>> signal
> > > > >>>>>>>>>>>> of project immaturity.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term in the
> > > long
> > > > >>>>>>>> run. I
> > > > >>>>>>>>>>> even
> > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the DAG
> > > > >> definition
> > > > >>>>>>>>>>> interface.
> > > > >>>>>>>>>>>> We better put our time and efforts in other places in
> > > Airflow,
> > > > >>>>>> of
> > > > >>>>>>>>> which
> > > > >>>>>>>>>>>> there are plenty.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Kind regards,
> > > > >>>>>>>>>>>> Igor
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > >>>>>>>>>> <bas@astronomer.io.inva <ma...@astronomer.io.inva>lid
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have already faded
> > > over
> > > > >>>>>>>> time,
> > > > >>>>>>>>>>>>> for example there are now several ways to create cyclic
> > > > >>>>>> graphs,
> > > > >>>>>>>>> e.g.
> > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these
> > properties
> > > > >>>>>>>>> vanishing
> > > > >>>>>>>>>>>>> even more in the future, so from that perspective I
> > support
> > > > >>>>>>>>> changing
> > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming:
> > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > >>>>>>>>>>>>> Prefect: flow
> > > > >>>>>>>>>>>>> Flyte: workflow
> > > > >>>>>>>>>>>>> Temporal: workflow
> > > > >>>>>>>>>>>>> Kestra: flow
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I suggest
> > > defining
> > > > >> a
> > > > >>>>>>>>> clear
> > > > >>>>>>>>>>>>> path forward. Would we first introduce the deprecation in
> > > > >>>>>>>> Airflow
> > > > >>>>>>>>> 3,
> > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Bas
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <neil4real@gmail.com
> > <ma...@gmail.com>>
> > > wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, especially when
> > > > >>>>>> most
> > > > >>>>>>>>> other
> > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing about it
> > at
> > > > >>>>>>>> all,
> > > > >>>>>>>>>>>>>> changing the term though would be fairly confusing to
> > most
> > > > >>>>>>>> users
> > > > >>>>>>>>>> who
> > > > >>>>>>>>>>>>>> have been
> > > > >>>>>>>>>>>>> using
> > > > >>>>>>>>>>>>>> the term for years.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung
> > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:tp@astronomer.io.inva
> > >lid
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term DAG. The
> > > only
> > > > >>>>>>>>>> problem
> > > > >>>>>>>>>>>>> (aside
> > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding myselfâto stop
> > > using
> > > > >>>>>>>> the
> > > > >>>>>>>>>>>>>>> term)
> > > > >>>>>>>>>>>>> is to
> > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned âworkflowâ is
> > > not
> > > > >>>>>>>> very
> > > > >>>>>>>>>>>>> accurate
> > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was proposed, but
> > > itâs a
> > > > >>>>>>>> bit
> > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and kept
> > finding
> > > > >>>>>>>> myself
> > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to clarify âDAG
> > > > >>>>>>>>>> definitionâ
> > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> TP
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > j_scheffler@gmx.de.INVA>LID>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same
> > > observation,
> > > > >>>>>>>> had a
> > > > >>>>>>>>>>>>>>>> short
> > > > >>>>>>>>>>>>>>> laight because the DAG question is always an
> > introduction
> > > > >>>>>> if
> > > > >>>>>>>>>>>>>>> someone
> > > > >>>>>>>>>>>>> joins
> > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes sense.
> > > > >>>>>> Especially
> > > > >>>>>>>>> when
> > > > >>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a reasonable step.
> > > > >>>>>>>> Concepts
> > > > >>>>>>>>>>>>>>> still
> > > > >>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>> stay the same.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding below the desk
> > > with
> > > > >>>>>>>> you
> > > > >>>>>>>>> and
> > > > >>>>>>>>>>>>>>>> +1
> > > > >>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep details of
> > > > >>>>>> python
> > > > >>>>>>>>> names
> > > > >>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>> same because the execution is still a DAG⊠but user
> > > facing
> > > > >>>>>> it
> > > > >>>>>>>>> is a
> > > > >>>>>>>>>>>>> workflow.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jens
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:ryan.hatter@astronomer.io
> > >
> > > > >>>>>>>>>>>>> .invalid>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at least for
> > > now.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to Airflow
> > > users.
> > > > >>>>>>>>> Indeed,
> > > > >>>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>> has
> > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians and
> > > > >>>>>> software
> > > > >>>>>>>>>>>>>>>>> engineers
> > > > >>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter. For
> > > someone
> > > > >>>>>>>> new
> > > > >>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, one of
> > the
> > > > >>>>>>>> first
> > > > >>>>>>>>>>>>>>>>> questions they
> > > > >>>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The answer
> > is
> > > > >>>>>>>> almost
> > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You don't
> > need
> > > to
> > > > >>>>>>>>> worry
> > > > >>>>>>>>>>>>>>>>> about what
> > > > >>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow."
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a couple
> > > > >>>>>>>> important
> > > > >>>>>>>>>>>> reasons:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned above, "DAG"
> > > is
> > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We want
> > > Airflow
> > > > >>>>>>>> to
> > > > >>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > >>>>>>>>>>>>>>> using
> > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat creates
> > > an
> > > > >>>>>>>>> initial
> > > > >>>>>>>>>>>>>>> barrier to
> > > > >>>>>>>>>>>>>>>>> understanding.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow Concepts*: The
> > DAG
> > > > >>>>>> is
> > > > >>>>>>>>> just
> > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The workflow
> > > > >>>>>> includes
> > > > >>>>>>>>> its
> > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other
> > parameters,
> > > > >>>>>> and
> > > > >>>>>>>>> other
> > > > >>>>>>>>>>>>>>>>> metadata that
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow homepage
> > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > https://airflow.apache.org/>>.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by the
> > community
> > > to
> > > > >>>>>>>>>>>>>>> programmatically
> > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" docs page
> > > > >>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>
> > > > >> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > https://airflow.apache.org/docs/apache-airflow/stable/index.html>
> > > > >>>>>>>>>>>>>>>>>> ,
> > > > >>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is without
> > using
> > > > >>>>>>>> "DAG."
> > > > >>>>>>>>>>>>>>>>> It's
> > > > >>>>>>>>>>>>>>> only in
> > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is
> > introduced
> > > > >>>>>> out
> > > > >>>>>>>> of
> > > > >>>>>>>>>>>>>>>>> nowhere
> > > > >>>>>>>>>>>>>>> as a
> > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection of tasks
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these
> > > > >>>>>> introductions
> > > > >>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate DAGs;
> > it
> > > > >>>>>>>>>>> orchestrates
> > > > >>>>>>>>>>>> workflows*.
> > > > >>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant to
> > almost
> > > > >>>>>>>> every
> > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for the term
> > > "DAG"
> > > > >>>>>>>> and
> > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with
> > > "workflow".
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> @dag(
> > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > >>>>>>>>>>>>>>>>> ...
> > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > >>>>>>>>>>>>>>>>> )
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > >>>>>>>>>>>>>>>>> ...
> > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > >>>>>>>>>>>>>>>>> )
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> And with that... I will start running away.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >> ------------------------------------------------------------------
> > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>> -------------------------------------------------------------------
> > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > >>>>>> dev-help@airflow.apache.org <mailto:dev-help@airflow.apache.org
> > >
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>
> > ---------------------------------------------------------------------
> > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org <mailto:
> > dev-unsubscribe@airflow.apache.org>
> > > > For additional commands, e-mail: dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>
> > > >
> > > >
> > >
> >
> >
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Airflow should deprecate the term "DAG" for end users
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
I stayed out of this discussion for the longest time, but I really try to
avoid "naming" discussions.
However, the extensions needed to Airflow for enhanced AI capabilities such
as cyclic operations and "human in the loop" workflows is something I have
been thinking a fair bit about. I also agree that these will need to come
relatively soon, before 3.5 :)
I am not advocating to reopen the discussion regarding naming, but wanted
to address one of the other segways in the discussion.
Vikram
On Tue, Feb 18, 2025 at 4:15âŻPM Jarek Potiuk <ja...@potiuk.com> wrote:
> Also, to hopefully at a little more humor to our "lively debate"
>
> > Iâm not sure conflating your opinion on this with âitâs just good
> engineering to make things blurry sometimesâ creates a logical truth.
>
> I think it really depends on the definition of "engineer", for me
> "engineer" is someone who builds stuff that is useful for people, not one
> that is tied to naming and classifications and mathematical abstractions
> (while the engineer knows and understands them perfectly well).
>
> This discussion reminded me of this (funny in my opinion) joke that
> illustrates the difference. I hope others will find it equally funny, and
> that I will not offend anyone with mathematical background (I apologise in
> advance if you feel offended):
>
> Two engineers who are flying a test flight of a new balloon they just built
> have been caught by a storm and they were lost in the middle of nowhere.
> They fly pretty low and they see a man walking down.
> They shout to him "Where are we now?!!"
> The man answers "In a balloon!!!!"
> The wind blows them far from the man and he disappears.
> Baffled for a moment, those two engineers look at each other and one of
> them says: "It must have been a mathematician."
> "Why?" asks the second.
> "Because his answer was extremely precise, and completely useless".
>
> Good night :)
>
> J.
>
>
>
>
>
>
> On Wed, Feb 19, 2025 at 12:49âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Yep. that will do as well for me - in the form proposed. We do not "have
> > to" redefine the name, leaving it as simply "dag" and explaining the
> origin
> > while clearly separating from it is also a good way to segway to
> "somewhat
> > cyclic" workflow / graph if we decide to (which I think will happen
> sooner
> > than 3.5 or 4.0 Jens :))
> >
> > J.
> >
> >
> > On Wed, Feb 19, 2025 at 12:42âŻAM Ryan Hatter
> > <ry...@astronomer.io.invalid> wrote:
> >
> >> Long after opening this can of worms, I also agree with Daniel S: Let's
> >> define "DAG" in the context of Airflow and be done with it (at least for
> >> now :) ). I've opened a docs PR attempting to do just that:
> >> https://github.com/apache/airflow/pull/46875
> >> <https://github.com/apache/airflow/pull/46875>
> >>
> >> On Tue, Feb 18, 2025 at 6:16âŻPM Ferruzzi, Dennis
> >> <fe...@amazon.com.invalid> wrote:
> >>
> >> > My two shillings: I came to Airflow knowing what a DAG was in the
> math
> >> > sense, and I was a bit surprised to see it used for Airflow. Our
> DAGS
> >> > aren't technically DAGs and haven't been since task retries were
> >> > introduced, maybe even before that. I'd support what Daniel said.
> IFF
> >> > we're going to change the name, I think "Workflow" works better than
> >> trying
> >> > to redefine an existing known term, but honestly I would advocate for
> >> > switching to using "Dag" as a proper noun with some little note
> >> somewhere
> >> > that the name comes from DAG but we've since evolved past that strict
> >> > definition.
> >> >
> >> >
> >> > - ferruzzi
> >> >
> >> >
> >> > ________________________________
> >> > From: Jens Scheffler <j_...@gmx.de.INVALID>
> >> > Sent: Tuesday, February 18, 2025 2:03 PM
> >> > To: dev@airflow.apache.org
> >> > Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end
> users
> >> >
> >> > CAUTION: This email originated from outside of the organization. Do
> not
> >> > click links or open attachments unless you can confirm the sender and
> >> know
> >> > the content is safe.
> >> >
> >> >
> >> >
> >> > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
> >> externe.
> >> > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne
> >> pouvez
> >> > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain
> >> que
> >> > le contenu ne présente aucun risque.
> >> >
> >> >
> >> >
> >> > Wow what a discussion thread. Was reading it and...:
> >> >
> >> > I am okay to clean up docs and agree to the others that we should NOT
> >> > change code interfaces.
> >> >
> >> > For the marketing part I need to repeat: (Almost) Everybody touching
> >> > Airflow needs an explanaition what "DAG" means. Changing the acronym
> to
> >> > have another meaning for DAG still needs an explanaition. For me
> >> > reanming the meaning of the Acronym does not bring any benefit, also
> not
> >> > Marketing.
> >> >
> >> > I have also seen multiple times challenges for teams in our area (even
> >> > outside ML) who wanted to iterate over results and we needed to
> >> > implement complexer multi-DAG structures to make this possible because
> >> > of DAG. If we would go in the direction as Jarek pitched (which might
> be
> >> > earliest 3.5 or 4.0) that a non-DAG workflow would be made possible
> (I'd
> >> > LOVE this!) then I would strongly opt for renaming it to "Workflow".
> >> > Because everybody understands (or thinks he understands) what it is
> and
> >> > DAG might be one implementation of this.
> >> >
> >> > So my conclusion is I am not for changing the meaning of DAG.
> >> >
> >> > On 18.02.25 19:54, Jarek Potiuk wrote:
> >> > > Ech. I would love so much if we could correct sent email same way we
> >> can
> >> > > correct messages in Slack :D
> >> > >
> >> > > On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
> >> > > <da...@astronomer.io.invalid> wrote:
> >> > >
> >> > >> damnit --- meant to say is *not* strictly speaking ....
> >> > >>
> >> > >> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
> >> > >> daniel.standish@astronomer.io> wrote:
> >> > >>
> >> > >>> Yeah I also disagree with code changes here. This thread went in
> an
> >> > >>> unexpected direction since I last poked my head in :)
> >> > >>>
> >> > >>> My thought is just in docs I would de-emphasize the mathy part of
> >> this.
> >> > >>> We can say a DAG is airflow's model for a collection of tasks that
> >> run,
> >> > >>> typically on a schedule. We could say further add, e.g. in *one
> >> place*
> >> > >>> somewhere, that the name DAG originated from a mathematical
> concept
> >> > >> called
> >> > >>> directed acyclic graph. But I do not think we need to go revising
> >> > >> history
> >> > >>> about that and providing new words for the acronym.
> >> > >>>
> >> > >>> But it has always been the reality that an Airflow DAG is strictly
> >> > >>> speaking a directed acyclic graph. It's something different. We
> do
> >> > >> forbid
> >> > >>> cycles. And it does contain the info needed to construct a graph
> of
> >> > the
> >> > >>> tasks. But it's much richer than that concept as well.
> >> > >>>
> >> > >>> I don't think we really need to go much further than that. But
> I'd
> >> > also
> >> > >>> be in support of writing `dag` or `dags` in docs instead of DAG or
> >> DAGs
> >> > >>> because it's ugly to do so, and unnecessary, and it invokes that
> >> mathy
> >> > >>> concept that is both confusing and inadequate.
> >> > >>>
> >> > >>>
> >> > >>>
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > For additional commands, e-mail: dev-help@airflow.apache.org
> >> >
> >> >
> >>
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Also, to hopefully at a little more humor to our "lively debate"
> Iâm not sure conflating your opinion on this with âitâs just good
engineering to make things blurry sometimesâ creates a logical truth.
I think it really depends on the definition of "engineer", for me
"engineer" is someone who builds stuff that is useful for people, not one
that is tied to naming and classifications and mathematical abstractions
(while the engineer knows and understands them perfectly well).
This discussion reminded me of this (funny in my opinion) joke that
illustrates the difference. I hope others will find it equally funny, and
that I will not offend anyone with mathematical background (I apologise in
advance if you feel offended):
Two engineers who are flying a test flight of a new balloon they just built
have been caught by a storm and they were lost in the middle of nowhere.
They fly pretty low and they see a man walking down.
They shout to him "Where are we now?!!"
The man answers "In a balloon!!!!"
The wind blows them far from the man and he disappears.
Baffled for a moment, those two engineers look at each other and one of
them says: "It must have been a mathematician."
"Why?" asks the second.
"Because his answer was extremely precise, and completely useless".
Good night :)
J.
On Wed, Feb 19, 2025 at 12:49âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> Yep. that will do as well for me - in the form proposed. We do not "have
> to" redefine the name, leaving it as simply "dag" and explaining the origin
> while clearly separating from it is also a good way to segway to "somewhat
> cyclic" workflow / graph if we decide to (which I think will happen sooner
> than 3.5 or 4.0 Jens :))
>
> J.
>
>
> On Wed, Feb 19, 2025 at 12:42âŻAM Ryan Hatter
> <ry...@astronomer.io.invalid> wrote:
>
>> Long after opening this can of worms, I also agree with Daniel S: Let's
>> define "DAG" in the context of Airflow and be done with it (at least for
>> now :) ). I've opened a docs PR attempting to do just that:
>> https://github.com/apache/airflow/pull/46875
>> <https://github.com/apache/airflow/pull/46875>
>>
>> On Tue, Feb 18, 2025 at 6:16âŻPM Ferruzzi, Dennis
>> <fe...@amazon.com.invalid> wrote:
>>
>> > My two shillings: I came to Airflow knowing what a DAG was in the math
>> > sense, and I was a bit surprised to see it used for Airflow. Our DAGS
>> > aren't technically DAGs and haven't been since task retries were
>> > introduced, maybe even before that. I'd support what Daniel said. IFF
>> > we're going to change the name, I think "Workflow" works better than
>> trying
>> > to redefine an existing known term, but honestly I would advocate for
>> > switching to using "Dag" as a proper noun with some little note
>> somewhere
>> > that the name comes from DAG but we've since evolved past that strict
>> > definition.
>> >
>> >
>> > - ferruzzi
>> >
>> >
>> > ________________________________
>> > From: Jens Scheffler <j_...@gmx.de.INVALID>
>> > Sent: Tuesday, February 18, 2025 2:03 PM
>> > To: dev@airflow.apache.org
>> > Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end users
>> >
>> > CAUTION: This email originated from outside of the organization. Do not
>> > click links or open attachments unless you can confirm the sender and
>> know
>> > the content is safe.
>> >
>> >
>> >
>> > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
>> externe.
>> > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne
>> pouvez
>> > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain
>> que
>> > le contenu ne présente aucun risque.
>> >
>> >
>> >
>> > Wow what a discussion thread. Was reading it and...:
>> >
>> > I am okay to clean up docs and agree to the others that we should NOT
>> > change code interfaces.
>> >
>> > For the marketing part I need to repeat: (Almost) Everybody touching
>> > Airflow needs an explanaition what "DAG" means. Changing the acronym to
>> > have another meaning for DAG still needs an explanaition. For me
>> > reanming the meaning of the Acronym does not bring any benefit, also not
>> > Marketing.
>> >
>> > I have also seen multiple times challenges for teams in our area (even
>> > outside ML) who wanted to iterate over results and we needed to
>> > implement complexer multi-DAG structures to make this possible because
>> > of DAG. If we would go in the direction as Jarek pitched (which might be
>> > earliest 3.5 or 4.0) that a non-DAG workflow would be made possible (I'd
>> > LOVE this!) then I would strongly opt for renaming it to "Workflow".
>> > Because everybody understands (or thinks he understands) what it is and
>> > DAG might be one implementation of this.
>> >
>> > So my conclusion is I am not for changing the meaning of DAG.
>> >
>> > On 18.02.25 19:54, Jarek Potiuk wrote:
>> > > Ech. I would love so much if we could correct sent email same way we
>> can
>> > > correct messages in Slack :D
>> > >
>> > > On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
>> > > <da...@astronomer.io.invalid> wrote:
>> > >
>> > >> damnit --- meant to say is *not* strictly speaking ....
>> > >>
>> > >> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
>> > >> daniel.standish@astronomer.io> wrote:
>> > >>
>> > >>> Yeah I also disagree with code changes here. This thread went in an
>> > >>> unexpected direction since I last poked my head in :)
>> > >>>
>> > >>> My thought is just in docs I would de-emphasize the mathy part of
>> this.
>> > >>> We can say a DAG is airflow's model for a collection of tasks that
>> run,
>> > >>> typically on a schedule. We could say further add, e.g. in *one
>> place*
>> > >>> somewhere, that the name DAG originated from a mathematical concept
>> > >> called
>> > >>> directed acyclic graph. But I do not think we need to go revising
>> > >> history
>> > >>> about that and providing new words for the acronym.
>> > >>>
>> > >>> But it has always been the reality that an Airflow DAG is strictly
>> > >>> speaking a directed acyclic graph. It's something different. We do
>> > >> forbid
>> > >>> cycles. And it does contain the info needed to construct a graph of
>> > the
>> > >>> tasks. But it's much richer than that concept as well.
>> > >>>
>> > >>> I don't think we really need to go much further than that. But I'd
>> > also
>> > >>> be in support of writing `dag` or `dags` in docs instead of DAG or
>> DAGs
>> > >>> because it's ugly to do so, and unnecessary, and it invokes that
>> mathy
>> > >>> concept that is both confusing and inadequate.
>> > >>>
>> > >>>
>> > >>>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Yep. that will do as well for me - in the form proposed. We do not "have
to" redefine the name, leaving it as simply "dag" and explaining the origin
while clearly separating from it is also a good way to segway to "somewhat
cyclic" workflow / graph if we decide to (which I think will happen sooner
than 3.5 or 4.0 Jens :))
J.
On Wed, Feb 19, 2025 at 12:42âŻAM Ryan Hatter
<ry...@astronomer.io.invalid> wrote:
> Long after opening this can of worms, I also agree with Daniel S: Let's
> define "DAG" in the context of Airflow and be done with it (at least for
> now :) ). I've opened a docs PR attempting to do just that:
> https://github.com/apache/airflow/pull/46875
> <https://github.com/apache/airflow/pull/46875>
>
> On Tue, Feb 18, 2025 at 6:16âŻPM Ferruzzi, Dennis
> <fe...@amazon.com.invalid> wrote:
>
> > My two shillings: I came to Airflow knowing what a DAG was in the math
> > sense, and I was a bit surprised to see it used for Airflow. Our DAGS
> > aren't technically DAGs and haven't been since task retries were
> > introduced, maybe even before that. I'd support what Daniel said. IFF
> > we're going to change the name, I think "Workflow" works better than
> trying
> > to redefine an existing known term, but honestly I would advocate for
> > switching to using "Dag" as a proper noun with some little note somewhere
> > that the name comes from DAG but we've since evolved past that strict
> > definition.
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Jens Scheffler <j_...@gmx.de.INVALID>
> > Sent: Tuesday, February 18, 2025 2:03 PM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end users
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur externe.
> > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne
> pouvez
> > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> > Wow what a discussion thread. Was reading it and...:
> >
> > I am okay to clean up docs and agree to the others that we should NOT
> > change code interfaces.
> >
> > For the marketing part I need to repeat: (Almost) Everybody touching
> > Airflow needs an explanaition what "DAG" means. Changing the acronym to
> > have another meaning for DAG still needs an explanaition. For me
> > reanming the meaning of the Acronym does not bring any benefit, also not
> > Marketing.
> >
> > I have also seen multiple times challenges for teams in our area (even
> > outside ML) who wanted to iterate over results and we needed to
> > implement complexer multi-DAG structures to make this possible because
> > of DAG. If we would go in the direction as Jarek pitched (which might be
> > earliest 3.5 or 4.0) that a non-DAG workflow would be made possible (I'd
> > LOVE this!) then I would strongly opt for renaming it to "Workflow".
> > Because everybody understands (or thinks he understands) what it is and
> > DAG might be one implementation of this.
> >
> > So my conclusion is I am not for changing the meaning of DAG.
> >
> > On 18.02.25 19:54, Jarek Potiuk wrote:
> > > Ech. I would love so much if we could correct sent email same way we
> can
> > > correct messages in Slack :D
> > >
> > > On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
> > > <da...@astronomer.io.invalid> wrote:
> > >
> > >> damnit --- meant to say is *not* strictly speaking ....
> > >>
> > >> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
> > >> daniel.standish@astronomer.io> wrote:
> > >>
> > >>> Yeah I also disagree with code changes here. This thread went in an
> > >>> unexpected direction since I last poked my head in :)
> > >>>
> > >>> My thought is just in docs I would de-emphasize the mathy part of
> this.
> > >>> We can say a DAG is airflow's model for a collection of tasks that
> run,
> > >>> typically on a schedule. We could say further add, e.g. in *one
> place*
> > >>> somewhere, that the name DAG originated from a mathematical concept
> > >> called
> > >>> directed acyclic graph. But I do not think we need to go revising
> > >> history
> > >>> about that and providing new words for the acronym.
> > >>>
> > >>> But it has always been the reality that an Airflow DAG is strictly
> > >>> speaking a directed acyclic graph. It's something different. We do
> > >> forbid
> > >>> cycles. And it does contain the info needed to construct a graph of
> > the
> > >>> tasks. But it's much richer than that concept as well.
> > >>>
> > >>> I don't think we really need to go much further than that. But I'd
> > also
> > >>> be in support of writing `dag` or `dags` in docs instead of DAG or
> DAGs
> > >>> because it's ugly to do so, and unnecessary, and it invokes that
> mathy
> > >>> concept that is both confusing and inadequate.
> > >>>
> > >>>
> > >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Ryan Hatter <ry...@astronomer.io.INVALID>.
Long after opening this can of worms, I also agree with Daniel S: Let's
define "DAG" in the context of Airflow and be done with it (at least for
now :) ). I've opened a docs PR attempting to do just that:
https://github.com/apache/airflow/pull/46875
<https://github.com/apache/airflow/pull/46875>
On Tue, Feb 18, 2025 at 6:16âŻPM Ferruzzi, Dennis
<fe...@amazon.com.invalid> wrote:
> My two shillings: I came to Airflow knowing what a DAG was in the math
> sense, and I was a bit surprised to see it used for Airflow. Our DAGS
> aren't technically DAGs and haven't been since task retries were
> introduced, maybe even before that. I'd support what Daniel said. IFF
> we're going to change the name, I think "Workflow" works better than trying
> to redefine an existing known term, but honestly I would advocate for
> switching to using "Dag" as a proper noun with some little note somewhere
> that the name comes from DAG but we've since evolved past that strict
> definition.
>
>
> - ferruzzi
>
>
> ________________________________
> From: Jens Scheffler <j_...@gmx.de.INVALID>
> Sent: Tuesday, February 18, 2025 2:03 PM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end users
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur externe.
> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne pouvez
> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> Wow what a discussion thread. Was reading it and...:
>
> I am okay to clean up docs and agree to the others that we should NOT
> change code interfaces.
>
> For the marketing part I need to repeat: (Almost) Everybody touching
> Airflow needs an explanaition what "DAG" means. Changing the acronym to
> have another meaning for DAG still needs an explanaition. For me
> reanming the meaning of the Acronym does not bring any benefit, also not
> Marketing.
>
> I have also seen multiple times challenges for teams in our area (even
> outside ML) who wanted to iterate over results and we needed to
> implement complexer multi-DAG structures to make this possible because
> of DAG. If we would go in the direction as Jarek pitched (which might be
> earliest 3.5 or 4.0) that a non-DAG workflow would be made possible (I'd
> LOVE this!) then I would strongly opt for renaming it to "Workflow".
> Because everybody understands (or thinks he understands) what it is and
> DAG might be one implementation of this.
>
> So my conclusion is I am not for changing the meaning of DAG.
>
> On 18.02.25 19:54, Jarek Potiuk wrote:
> > Ech. I would love so much if we could correct sent email same way we can
> > correct messages in Slack :D
> >
> > On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
> > <da...@astronomer.io.invalid> wrote:
> >
> >> damnit --- meant to say is *not* strictly speaking ....
> >>
> >> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
> >> daniel.standish@astronomer.io> wrote:
> >>
> >>> Yeah I also disagree with code changes here. This thread went in an
> >>> unexpected direction since I last poked my head in :)
> >>>
> >>> My thought is just in docs I would de-emphasize the mathy part of this.
> >>> We can say a DAG is airflow's model for a collection of tasks that run,
> >>> typically on a schedule. We could say further add, e.g. in *one place*
> >>> somewhere, that the name DAG originated from a mathematical concept
> >> called
> >>> directed acyclic graph. But I do not think we need to go revising
> >> history
> >>> about that and providing new words for the acronym.
> >>>
> >>> But it has always been the reality that an Airflow DAG is strictly
> >>> speaking a directed acyclic graph. It's something different. We do
> >> forbid
> >>> cycles. And it does contain the info needed to construct a graph of
> the
> >>> tasks. But it's much richer than that concept as well.
> >>>
> >>> I don't think we really need to go much further than that. But I'd
> also
> >>> be in support of writing `dag` or `dags` in docs instead of DAG or DAGs
> >>> because it's ugly to do so, and unnecessary, and it invokes that mathy
> >>> concept that is both confusing and inadequate.
> >>>
> >>>
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by "Ferruzzi, Dennis" <fe...@amazon.com.INVALID>.
My two shillings: I came to Airflow knowing what a DAG was in the math sense, and I was a bit surprised to see it used for Airflow. Our DAGS aren't technically DAGs and haven't been since task retries were introduced, maybe even before that. I'd support what Daniel said. IFF we're going to change the name, I think "Workflow" works better than trying to redefine an existing known term, but honestly I would advocate for switching to using "Dag" as a proper noun with some little note somewhere that the name comes from DAG but we've since evolved past that strict definition.
- ferruzzi
________________________________
From: Jens Scheffler <j_...@gmx.de.INVALID>
Sent: Tuesday, February 18, 2025 2:03 PM
To: dev@airflow.apache.org
Subject: RE: [EXT] Airflow should deprecate the term "DAG" for end users
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur externe. Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne pouvez pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain que le contenu ne prĂ©sente aucun risque.
Wow what a discussion thread. Was reading it and...:
I am okay to clean up docs and agree to the others that we should NOT
change code interfaces.
For the marketing part I need to repeat: (Almost) Everybody touching
Airflow needs an explanaition what "DAG" means. Changing the acronym to
have another meaning for DAG still needs an explanaition. For me
reanming the meaning of the Acronym does not bring any benefit, also not
Marketing.
I have also seen multiple times challenges for teams in our area (even
outside ML) who wanted to iterate over results and we needed to
implement complexer multi-DAG structures to make this possible because
of DAG. If we would go in the direction as Jarek pitched (which might be
earliest 3.5 or 4.0) that a non-DAG workflow would be made possible (I'd
LOVE this!) then I would strongly opt for renaming it to "Workflow".
Because everybody understands (or thinks he understands) what it is and
DAG might be one implementation of this.
So my conclusion is I am not for changing the meaning of DAG.
On 18.02.25 19:54, Jarek Potiuk wrote:
> Ech. I would love so much if we could correct sent email same way we can
> correct messages in Slack :D
>
> On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
> <da...@astronomer.io.invalid> wrote:
>
>> damnit --- meant to say is *not* strictly speaking ....
>>
>> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
>> daniel.standish@astronomer.io> wrote:
>>
>>> Yeah I also disagree with code changes here. This thread went in an
>>> unexpected direction since I last poked my head in :)
>>>
>>> My thought is just in docs I would de-emphasize the mathy part of this.
>>> We can say a DAG is airflow's model for a collection of tasks that run,
>>> typically on a schedule. We could say further add, e.g. in *one place*
>>> somewhere, that the name DAG originated from a mathematical concept
>> called
>>> directed acyclic graph. But I do not think we need to go revising
>> history
>>> about that and providing new words for the acronym.
>>>
>>> But it has always been the reality that an Airflow DAG is strictly
>>> speaking a directed acyclic graph. It's something different. We do
>> forbid
>>> cycles. And it does contain the info needed to construct a graph of the
>>> tasks. But it's much richer than that concept as well.
>>>
>>> I don't think we really need to go much further than that. But I'd also
>>> be in support of writing `dag` or `dags` in docs instead of DAG or DAGs
>>> because it's ugly to do so, and unnecessary, and it invokes that mathy
>>> concept that is both confusing and inadequate.
>>>
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Wow what a discussion thread. Was reading it and...:
I am okay to clean up docs and agree to the others that we should NOT
change code interfaces.
For the marketing part I need to repeat: (Almost) Everybody touching
Airflow needs an explanaition what "DAG" means. Changing the acronym to
have another meaning for DAG still needs an explanaition. For me
reanming the meaning of the Acronym does not bring any benefit, also not
Marketing.
I have also seen multiple times challenges for teams in our area (even
outside ML) who wanted to iterate over results and we needed to
implement complexer multi-DAG structures to make this possible because
of DAG. If we would go in the direction as Jarek pitched (which might be
earliest 3.5 or 4.0) that a non-DAG workflow would be made possible (I'd
LOVE this!) then I would strongly opt for renaming it to "Workflow".
Because everybody understands (or thinks he understands) what it is and
DAG might be one implementation of this.
So my conclusion is I am not for changing the meaning of DAG.
On 18.02.25 19:54, Jarek Potiuk wrote:
> Ech. I would love so much if we could correct sent email same way we can
> correct messages in Slack :D
>
> On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
> <da...@astronomer.io.invalid> wrote:
>
>> damnit --- meant to say is *not* strictly speaking ....
>>
>> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
>> daniel.standish@astronomer.io> wrote:
>>
>>> Yeah I also disagree with code changes here. This thread went in an
>>> unexpected direction since I last poked my head in :)
>>>
>>> My thought is just in docs I would de-emphasize the mathy part of this.
>>> We can say a DAG is airflow's model for a collection of tasks that run,
>>> typically on a schedule. We could say further add, e.g. in *one place*
>>> somewhere, that the name DAG originated from a mathematical concept
>> called
>>> directed acyclic graph. But I do not think we need to go revising
>> history
>>> about that and providing new words for the acronym.
>>>
>>> But it has always been the reality that an Airflow DAG is strictly
>>> speaking a directed acyclic graph. It's something different. We do
>> forbid
>>> cycles. And it does contain the info needed to construct a graph of the
>>> tasks. But it's much richer than that concept as well.
>>>
>>> I don't think we really need to go much further than that. But I'd also
>>> be in support of writing `dag` or `dags` in docs instead of DAG or DAGs
>>> because it's ugly to do so, and unnecessary, and it invokes that mathy
>>> concept that is both confusing and inadequate.
>>>
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Ech. I would love so much if we could correct sent email same way we can
correct messages in Slack :D
On Tue, Feb 18, 2025 at 7:50âŻPM Daniel Standish
<da...@astronomer.io.invalid> wrote:
> damnit --- meant to say is *not* strictly speaking ....
>
> On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
> daniel.standish@astronomer.io> wrote:
>
> > Yeah I also disagree with code changes here. This thread went in an
> > unexpected direction since I last poked my head in :)
> >
> > My thought is just in docs I would de-emphasize the mathy part of this.
> > We can say a DAG is airflow's model for a collection of tasks that run,
> > typically on a schedule. We could say further add, e.g. in *one place*
> > somewhere, that the name DAG originated from a mathematical concept
> called
> > directed acyclic graph. But I do not think we need to go revising
> history
> > about that and providing new words for the acronym.
> >
> > But it has always been the reality that an Airflow DAG is strictly
> > speaking a directed acyclic graph. It's something different. We do
> forbid
> > cycles. And it does contain the info needed to construct a graph of the
> > tasks. But it's much richer than that concept as well.
> >
> > I don't think we really need to go much further than that. But I'd also
> > be in support of writing `dag` or `dags` in docs instead of DAG or DAGs
> > because it's ugly to do so, and unnecessary, and it invokes that mathy
> > concept that is both confusing and inadequate.
> >
> >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
damnit --- meant to say is *not* strictly speaking ....
On Tue, Feb 18, 2025 at 10:46âŻAM Daniel Standish <
daniel.standish@astronomer.io> wrote:
> Yeah I also disagree with code changes here. This thread went in an
> unexpected direction since I last poked my head in :)
>
> My thought is just in docs I would de-emphasize the mathy part of this.
> We can say a DAG is airflow's model for a collection of tasks that run,
> typically on a schedule. We could say further add, e.g. in *one place*
> somewhere, that the name DAG originated from a mathematical concept called
> directed acyclic graph. But I do not think we need to go revising history
> about that and providing new words for the acronym.
>
> But it has always been the reality that an Airflow DAG is strictly
> speaking a directed acyclic graph. It's something different. We do forbid
> cycles. And it does contain the info needed to construct a graph of the
> tasks. But it's much richer than that concept as well.
>
> I don't think we really need to go much further than that. But I'd also
> be in support of writing `dag` or `dags` in docs instead of DAG or DAGs
> because it's ugly to do so, and unnecessary, and it invokes that mathy
> concept that is both confusing and inadequate.
>
>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Daniel Standish <da...@astronomer.io.INVALID>.
Yeah I also disagree with code changes here. This thread went in an
unexpected direction since I last poked my head in :)
My thought is just in docs I would de-emphasize the mathy part of this. We
can say a DAG is airflow's model for a collection of tasks that run,
typically on a schedule. We could say further add, e.g. in *one place*
somewhere, that the name DAG originated from a mathematical concept called
directed acyclic graph. But I do not think we need to go revising history
about that and providing new words for the acronym.
But it has always been the reality that an Airflow DAG is strictly speaking
a directed acyclic graph. It's something different. We do forbid cycles.
And it does contain the info needed to construct a graph of the tasks. But
it's much richer than that concept as well.
I don't think we really need to go much further than that. But I'd also be
in support of writing `dag` or `dags` in docs instead of DAG or DAGs
because it's ugly to do so, and unnecessary, and it invokes that mathy
concept that is both confusing and inadequate.
Re: Airflow should deprecate the term "DAG" for end users
Posted by Brian Greene <br...@heisenbergwoodworking.com>.
Iâm not sure conflating your opinion on this with âitâs just good
engineering to make things blurry sometimesâ creates a logical truth.
:-)
On Tue, Feb 18, 2025 at 12:41âŻPM Jarek Potiuk <ja...@potiuk.com> wrote:
> Oh no.. Swords, swords, .... :)
>
> Hello Brian ! Nice to see you back in the conversations. We've been
> discussing things like this before a few times even over the phone, I
> recall.
>
> Yes, I think it's partially marketing, I agree. And marketing that is
> important for the project, which we as engineers, should also at least
> understand. We should worry about how our work is perceived by
> non-engineers or at least not-so-much-deep-into-technical-correctness
> engineers who just want to get stuff done. This is at least my goal of
> "communicating" such stuff through documentation - recognise that the vast
> majority of our users just want to get things done and don't really care
> about abstract mathematical concepts, they might not even be aware of. Our
> role in this project is not to teach people math or algorithmics, but to
> help them do their job. That's our primary goal.
>
> And to that - more importantly (for the reasons i and Constance explained)
> we **might** soon not have "technically Directed Acyclic Graphs" - we do
> not know if we go that route, but IMHO, it's likely. Then from a good
> engineering point of view, figuring a better name that would ease the
> transition is not only smart, but also fair, and it will make our life
> easier, I am not a fan of being "100% precise and correct" if it makes my
> future life harder. As an experienced engineer I am happy to go down the
> path of making definitions more blurry, if it is still "good definition" -
> fitting the purpose, and making it easier to transition to the place where
> the old definition will no longer be true.
>
> That's what I consider being a great engineer is - to not only focus on
> minute details and definition correctness, but also understanding of my
> users, communicating with them in the way that makes it easier for them to
> relate to the project and "do their job", and to foresee and clear the path
> for future evolution of my product.
>
> I hope this explanation can provide a bit more context - at least how I
> understand it, and I hope you will see it's not a "because we want it"
> change. It has some real grounding, reasoning, expectations and is
> something that few of us have been thinking about for a while. So please do
> not see it as "marketing driven" change. For me it's "engineering driven
> change that takes into account marketing".
>
> J.
>
>
> On Tue, Feb 18, 2025 at 7:20âŻPM Brian Greene <
> brian@heisenbergwoodworking.com> wrote:
>
> > I disagree with renaming it, or removing the logical idea of directed
> > acyclic graph, or redefining the acronym. Itâs less âmathâ to me, and
> more
> > âdata structureâ, and a lot of things work the way they do because a dag
> IS
> > a DAG.
> >
> > Whatâs the problem with calling something EXACTLY what it is? So rarely
> > does a technical team get it so right⊠this feels like a discussion with
> > marketing people, wanting to obfuscate something completely reasonable,
> for
> > no functional value.
> >
> > How much energy can be spent making this scenario more common? âwhatâs
> DAG
> > vs Dag vs Oh you mean DaaG :-(⊠or whateverâ. And why would one want to?
> >
> > Brian
> >
> > On Tue, Feb 18, 2025 at 11:45âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > -0.5
> > > > Even though the new UI prefers "Dag", I don't think that means "DAG"
> > is
> > > incorrect.
> > >
> > > I think we have a bunch of related things discussed here - I am not
> > exactly
> > > sure what the -0.5 in this context means Brent. Would it be possible to
> > > clarify ?
> > >
> > > On Tue, Feb 18, 2025 at 6:01âŻPM Ambika Garg <am...@gmail.com>
> > > wrote:
> > >
> > > > +1 would vote for
> > > > Airflow's *D*ynamic *A*ction *G*raph (DAG)
> > > >
> > > > *Dynamic*: Reflects the capability of Airflow to handle workflows
> that
> > > > arenât strictly acyclic, but can adapt to changes, iterations and
> loops
> > > > within broader context of workflow.
> > > >
> > > > *Action*: Emphasizes that airflow is fundamentally about
> orchestrating
> > > > actions or tasks. Whether its data processing, model training or any
> > > other
> > > > form of computation. It also conveys operational aspect of what
> Airflow
> > > > does.
> > > >
> > > > *Graph*: Maintains the core concept that workflows in Airflow are
> > > > visualized and managed as graphs. This term keeps the idea of
> > > dependencies
> > > > and relationships between tasks intact.
> > > >
> > > > Ambika.
> > > >
> > > > On Tue, Feb 18, 2025 at 11:46âŻAM Brent Bovenzi
> > > <brent@astronomer.io.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > -0.5
> > > > >
> > > > > Even though the new UI prefers "Dag", I don't think that means
> "DAG"
> > is
> > > > > incorrect.
> > > > >
> > > > > On Tue, Feb 18, 2025 at 11:36âŻAM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >
> > > > > > I love "Dynamic Airflow Graph" +10 to that Constance.
> > > > > >
> > > > > > And yes. I fully agree with your "somewhat" acyclic statement. I
> > saw
> > > > > people
> > > > > > mentioning "event-driven" workflows to better describe the AI/ML,
> > but
> > > > > that
> > > > > > for me is a synonym of "completely unmanageable workflows"
> (similar
> > > to
> > > > > > micro-services) - what I was alluding to was exactly this -
> > something
> > > > > like
> > > > > > curren DAG with parts of it being potentially cyclic - so we are
> > > > > completely
> > > > > > aligned here :). And that's what make it "not DAG - in
> mathematical
> > > > > sense"
> > > > > > - even more reason to depart from it,
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
> > > > > > <co...@astronomer.io.invalid> wrote:
> > > > > >
> > > > > > > Hey all,
> > > > > > >
> > > > > > > I appreciate the discussion about how we position the concept
> of
> > a
> > > > DAG
> > > > > in
> > > > > > > Airflow. While I agree with updating documentation to
> > de-emphasize
> > > > the
> > > > > > > strict mathematical definition, I do not think we should change
> > the
> > > > > > public
> > > > > > > interface to use "dag" instead of "DAG" everywhere. It adds
> > > > > > > unnecessary churn and breaks consistency across the ecosystem.
> > > > > > >
> > > > > > > Regarding the broader discussion on whether DAGs in Airflow
> > should
> > > > > remain
> > > > > > > acyclic: I acknowledge that some workflows, particularly in ML
> > and
> > > > AI,
> > > > > > > require cyclical behaviour (example: iterative refinement
> until a
> > > > > > condition
> > > > > > > is met). However, I think it's important to recognize that even
> > in
> > > > > those
> > > > > > > cases, the workflow itself still has an overall start and end.
> > It's
> > > > not
> > > > > > an
> > > > > > > unbounded loop but rather a controlled execution process that
> > > happens
> > > > > to
> > > > > > > have cycles *within* it.
> > > > > > >
> > > > > > > That being said, I do see value in reinterpreting "DAG" to
> better
> > > > align
> > > > > > > with how Airflow is actually used, and love shifting away from
> > > > > "Directed
> > > > > > > Acyclic Graph" in favour of something more intuitive. However
> > "Data
> > > > > > Airflow
> > > > > > > Graph" makes it sound like we're defining Airflow itself rather
> > > than
> > > > > > > describing the structure of workflows within it.
> > > > > > >
> > > > > > > If we want a new meaning for DAG that resonates with our base
> > while
> > > > > > keeping
> > > > > > > Airflow's identity clear, I am proposing three alternatives:
> > > > > > >
> > > > > > >
> > > > > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while
> > > > > emphasizing
> > > > > > > flexibility
> > > > > > > 2. *Distributed Automation Graph*: Highlights execution
> across
> > > > > systems
> > > > > > > and automations without tying it strictly to data
> > > > > > > 3. *Directed Activity Graph*: Retains the useful meaning of
> > > > > "Directed"
> > > > > > > while shifting away from strict acyclic constraints. (My
> > > personal
> > > > > > > favourite)
> > > > > > >
> > > > > > > I think any shift in terminology should be driven by what best
> > > > > > communicates
> > > > > > > Airflow's role workflow automation, rather than just distancing
> > > > > ourselves
> > > > > > > from the mathematical definition.
> > > > > > >
> > > > > > > Curious to hear more thoughts!
> > > > > > >
> > > > > > > On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <
> jarek@potiuk.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Sorry for the few typos ... My slight dyslexia did not help
> and
> > > > > > > autocorrect
> > > > > > > > did not help either this time : dept -> debt, routed ->
> rooted
> > > and
> > > > a
> > > > > > few
> > > > > > > > other small typos :) .
> > > > > > > >
> > > > > > > > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <
> jarek@potiuk.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > First of all - yes, agree with Lee and Ash, I think we do
> not
> > > > need
> > > > > to
> > > > > > > > > change the public interface, I'd also be -0 on that, but
> > docs,
> > > > > > internal
> > > > > > > > > strings, yes we could.
> > > > > > > > >
> > > > > > > > > But .... After thinking a bit and looking at the discussion
> > > here,
> > > > > > where
> > > > > > > > > people have concerns, I have an alternative take. And I
> know
> > > this
> > > > > > might
> > > > > > > > be
> > > > > > > > > controversial, and some people might find it confusing, but
> > why
> > > > > don't
> > > > > > > we
> > > > > > > > > come up with a new acronym for DAG, one that will be more
> > > airflow
> > > > > and
> > > > > > > > less
> > > > > > > > > "mathematics/algorithm". And I have, I think a good reason
> > for
> > > > it,
> > > > > > and
> > > > > > > I
> > > > > > > > > even have a proposal. Bear with me,
> > > > > > > > >
> > > > > > > > > DAG => *Data Airflow Graph*
> > > > > > > > >
> > > > > > > > > As many of us mentioned, DAG and Airflow are almost 100%
> 1-1
> > > now
> > > > > > (with
> > > > > > > > > exception of some niche products - you know which one I
> speak
> > > > > about -
> > > > > > > > which
> > > > > > > > > are unfortunate enough to use the "Dag" in their name - I
> > will
> > > > > > explain
> > > > > > > > why
> > > > > > > > > this is unfortunate, IMHO).
> > > > > > > > >
> > > > > > > > > We are lucky to have Airflow for "A" in the acronym and we
> > can
> > > > make
> > > > > > it
> > > > > > > > > even more "synonymous" -> Airflow <> DAG with (A) being
> > > Airflow.
> > > > > > > > >
> > > > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > > > > > > > >
> > > > > > > > > Not only because "Directed" and "Acyclic" are mostly not in
> > the
> > > > > > > > vocabulary
> > > > > > > > > of most of our users - as Ryan, starting this thread
> > explained
> > > > > yes, I
> > > > > > > > agree
> > > > > > > > > with it, but also I think there is a more important reason
> -
> > I
> > > > > > simply
> > > > > > > > > believe in the future our DAGs might not (and likely will
> > not)
> > > be
> > > > > > > > "acyclic".
> > > > > > > > >
> > > > > > > > > I know for some it might be herezy and we have not
> discussed
> > > any
> > > > of
> > > > > > > that
> > > > > > > > > yet, but more and more I keep on hearing that Machine
> > Learning
> > > > > > > Workflows
> > > > > > > > of
> > > > > > > > > the future are often somewhat cyclic. In our case there is
> > one
> > > > type
> > > > > > of
> > > > > > > > > repeat-in-cycle behaviour - we have "task retry on
> failure" -
> > > but
> > > > > > this
> > > > > > > is
> > > > > > > > > only on failure and it basically mens "idempotent" retry -
> we
> > > > want
> > > > > to
> > > > > > > > redo
> > > > > > > > > something and get single result out of that and basically
> > > discard
> > > > > the
> > > > > > > > > previous result. But machine learning/AI workflows are
> > > > different. I
> > > > > > > had a
> > > > > > > > > lot of conversations recently (I even co-organized "AI
> > > low-level
> > > > > > > > > engineering and hacking" devroom to learn more about
> those) -
> > > > and I
> > > > > > > keep
> > > > > > > > on
> > > > > > > > > hearing that "Repeat a sequence of things several times and
> > > > refine"
> > > > > > is
> > > > > > > > > precisely what AI / ML workflows will need - especially
> with
> > > > > Agentic
> > > > > > > > > workflows - often we have cooperating agents which
> **could**
> > be
> > > > > part
> > > > > > of
> > > > > > > > the
> > > > > > > > > same DAG, but currently in Airflow where our DAGs are
> > > "Acyclic"
> > > > we
> > > > > > can
> > > > > > > > > only express "repeat several time and refine" as a single
> > task
> > > > that
> > > > > > > will
> > > > > > > > do
> > > > > > > > > such refining. If we would like to split such an "agent"
> > doing
> > > > > their
> > > > > > > own
> > > > > > > > > work in "cycles" and communicating with other "agents"
> doing
> > > the
> > > > > > same -
> > > > > > > > > represented as a multi-task-entity, we have no way to
> express
> > > it
> > > > > > using
> > > > > > > > our
> > > > > > > > > current "acyclic" semantics.
> > > > > > > > >
> > > > > > > > > But - other than our implementation makes that assumption
> > > > > currently -
> > > > > > > > > there is nothing fundamental with Airflow being able to run
> > > such
> > > > > > > > "partially
> > > > > > > > > cyclic workflows". We have all the pieces, we would have to
> > > > change
> > > > > > > > > dependency calculation to allow for cycles, update UI to
> > handle
> > > > > > cycles
> > > > > > > > > representation - and we could in theory quite seamlessly,
> > > support
> > > > > > > > "cyclic"
> > > > > > > > > workflows.
> > > > > > > > >
> > > > > > > > > I know, I know, that's the first time we hear it and that's
> > > huge
> > > > > > > change -
> > > > > > > > > of course, I do not want to start discussing any details of
> > it,
> > > > not
> > > > > > > > before
> > > > > > > > > Airflow 3.0 is out - but I think we should be open that at
> > some
> > > > > point
> > > > > > > of
> > > > > > > > > time the "Acyclic" property is going to go away, And my
> > > personal
> > > > > > > > intuition
> > > > > > > > > tells me that a year from now we will have acyclic
> workflows
> > > > > > (assuming
> > > > > > > > that
> > > > > > > > > all the foundational work we do on Airflow 3 will really
> help
> > > in
> > > > > > > > decreasing
> > > > > > > > > all the technical dept, and will open our minds to new
> > ideas).
> > > > > > > > >
> > > > > > > > > That's why I think - the sooner we get rid of "Directed
> > > Acyclic"
> > > > -
> > > > > > the
> > > > > > > > > better. that will - even for us, maintainer free us from
> > being
> > > > > routed
> > > > > > > in
> > > > > > > > > the acyclic nature of Airflow.
> > > > > > > > >
> > > > > > > > > How about we change to "Data Airflow Graph" (or any other
> > good
> > > > > > acronym
> > > > > > > > > which has (A)irflow instead of (A)cyclic). The name is
> just a
> > > > > > proposal.
> > > > > > > > >
> > > > > > > > > Now.. I go hide somewhere as I believe that might cause a
> > > > > "flamewar"
> > > > > > > :),
> > > > > > > > > unless on request of Ryan you already sheathed your swords
> in
> > > the
> > > > > > > > > discussion (which seems everyone did).
> > > > > > > > >
> > > > > > > > > But honestly - I would really love what you think of "let's
> > > just
> > > > > > break
> > > > > > > > > with Acyclic NOW" and shape our future by using a different
> > > > > acronym.
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > > > > > > > ephraimanierobi@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > >> When I started contributing to Airflow, I had to read up
> on
> > > DAG,
> > > > > and
> > > > > > > > now I
> > > > > > > > >> know what it means. I wonder if we are about to have users
> > who
> > > > use
> > > > > > the
> > > > > > > > >> term
> > > > > > > > >> Dag without knowing that there's a DAG.
> > > > > > > > >>
> > > > > > > > >> Slightly concerned.
> > > > > > > > >>
> > > > > > > > >> -ephraim
> > > > > > > > >>
> > > > > > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <
> weilee.rx@gmail.com>
> > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > For user-facing things (e.g., `from airflow import
> DAG`),
> > > Iâm
> > > > > more
> > > > > > > > like
> > > > > > > > >> > -0. But for documentation, docstring, internal things
> and
> > > > etc.,
> > > > > we
> > > > > > > > >> probably
> > > > > > > > >> > could still change most of them?
> > > > > > > > >> >
> > > > > > > > >> > Best,
> > > > > > > > >> > Wei
> > > > > > > > >> >
> > > > > > > > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <
> > > > > ash@apache.org>
> > > > > > > > >> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > I also donât personally think itâs worth the pain (not
> > to
> > > > > > mention
> > > > > > > > >> > backcompat workaround) to rename DAG to Dag, so Iâd be
> > -0.5
> > > on
> > > > > > that.
> > > > > > > > >> > >
> > > > > > > > >> > > -ash
> > > > > > > > >> > >
> > > > > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > > > > > > > daniel.imberman@gmail.com
> > > > > > > > >> >
> > > > > > > > >> > wrote:
> > > > > > > > >> > >>
> > > > > > > > >> > >> I think my biggest concern is a marketing one and
> not a
> > > > > > technical
> > > > > > > > >> one.
> > > > > > > > >> > >>
> > > > > > > > >> > >> As has been mentioned on the thread the terms airflow
> > and
> > > > dag
> > > > > > are
> > > > > > > > >> kind
> > > > > > > > >> > of
> > > > > > > > >> > >> synonymous and I certainly donât want to give the
> > > > impression
> > > > > > that
> > > > > > > > we
> > > > > > > > >> are
> > > > > > > > >> > >> breaking more than we are breaking.
> > > > > > > > >> > >>
> > > > > > > > >> > >> I wouldnât die on this hill, but Iâm slightly
> > concerned.
> > > > > > > > >> > >>
> > > > > > > > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <
> > > weilee@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >> > >>
> > > > > > > > >> > >>> Iâm not sure about adding ruff rules here đ€ I think
> > > ruff
> > > > > > rules
> > > > > > > > are
> > > > > > > > >> > best
> > > > > > > > >> > >>> suited for user-facing things but not the airflow
> code
> > > > base
> > > > > > > > itself.
> > > > > > > > >> If
> > > > > > > > >> > what
> > > > > > > > >> > >>> we mean is adding a rule to avoid users using "DAG"
> > > > *after*
> > > > > we
> > > > > > > > >> rename
> > > > > > > > >> > it,
> > > > > > > > >> > >>> it's definitely a +1000.
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> I just created GitHub issues for this removing "DAG"
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> https://github.com/apache/airflow/issues/46842
> > > > > > > > >> > >>> https://github.com/apache/airflow/issues/46843
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> One thing I'm not sure about is whether we want to
> get
> > > rid
> > > > > of
> > > > > > > > `from
> > > > > > > > >> > >>> airflow import DAG` as well. đ€
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> Best,
> > > > > > > > >> > >>> Wei
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > > > > > > > >> > >>>> Hard to say until it's looked at :)
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > > > > > > > >> aritrabasu1999@gmail.com
> > > > > > > > >> > >
> > > > > > > > >> > >>>> wrote:
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>>> I can take it up, it's mostly just a doc update
> > right?
> > > > Or
> > > > > > are
> > > > > > > we
> > > > > > > > >> > doing
> > > > > > > > >> > >>> code
> > > > > > > > >> > >>>>> files replacement too?
> > > > > > > > >> > >>>>> --
> > > > > > > > >> > >>>>> Regards,
> > > > > > > > >> > >>>>> Aritra Basu
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <
> > > > > > jarek@potiuk.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>>>> Sounds like another +1000 files big PR is coming
> > :) ?
> > > > Any
> > > > > > > > >> volunteers
> > > > > > > > >> > >>> to
> > > > > > > > >> > >>>>>> make it ? It's fun.
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > > > > > > > droiddev510e@gmail.com>
> > > > > > > > >> > >>> wrote:
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>>> +1 for ruff rules :)
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace
> > > 'DAG'
> > > > > in
> > > > > > > the
> > > > > > > > >> > >>> Airflow
> > > > > > > > >> > >>>>>>> docs, in line with the new UI changes and to
> make
> > > the
> > > > > > > renaming
> > > > > > > > >> > >>>>>>> consistent across user-facing pages.
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> Regards,
> > > > > > > > >> > >>>>>>> Omkar
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > > > > > > > jarek@potiuk.com>
> > > > > > > > >> > >>> wrote:
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that
> > :)
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <
> > > > > > weilee@apache.org
> > > > > > > >
> > > > > > > > >> > >>> wrote:
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>>> It seems that our current conclusion is to use
> > > "dag"
> > > > > or
> > > > > > > > "Dag"
> > > > > > > > >> > >>>>> instead
> > > > > > > > >> > >>>>>>> of
> > > > > > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all
> > > "DAG"
> > > > > in
> > > > > > > the
> > > > > > > > >> > >>>>> codebase
> > > > > > > > >> > >>>>>>> with
> > > > > > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that
> (which
> > > it
> > > > > > might
> > > > > > > be
> > > > > > > > >> > >>> đ€),
> > > > > > > > >> > >>>>>>> should
> > > > > > > > >> > >>>>>>>> we
> > > > > > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the
> > > > following
> > > > > > PRs?
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> Best,
> > > > > > > > >> > >>>>>>>>> Wei
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new
> > 3.0
> > > > UI:
> > > > > > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > > > > > > > >> > >>>>>>>>>> Let me know if that's the direction we want
> to
> > > go.
> > > > > > > > >> > >>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo,
> > > > Rajeshwar
> > > > > > > > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > > > > > > > >> > >>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best,
> > "dag"
> > > > is
> > > > > > > > >> > >>> synonymous
> > > > > > > > >> > >>>>>> with
> > > > > > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk
> > and
> > > > > > explain
> > > > > > > > >> > >>> what
> > > > > > > > >> > >>>>>>> Airflow
> > > > > > > > >> > >>>>>>>>> is
> > > > > > > > >> > >>>>>>>>>>> all about. Although folks would ask, I donât
> > > ever
> > > > > use
> > > > > > > the
> > > > > > > > >> > >>>>>>>> mathematical
> > > > > > > > >> > >>>>>>>>>>> definition of DAG.
> > > > > > > > >> > >>>>>>>>>>> It will be challenging and possibly
> confusing
> > > for
> > > > > many
> > > > > > > > >> > >>> users
> > > > > > > > >> > >>>>>> making
> > > > > > > > >> > >>>>>>>>> such a
> > > > > > > > >> > >>>>>>>>>>> change - I would rather direct that energy
> to
> > > > > > appending
> > > > > > > > the
> > > > > > > > >> > >>>>>> Oxford
> > > > > > > > >> > >>>>>>>>>>> definition of "dag" to include a reference
> to
> > > > > > workflows
> > > > > > > in
> > > > > > > > >> > >>>>>> Airflow.
> > > > > > > > >> > >>>>>>>> __
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> -- Rajesh
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> > > > > > > jarek@potiuk.com
> > > > > > > > >> > >>>>>> <mailto:
> > > > > > > > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside
> of
> > > the
> > > > > > > > >> > >>>>> organization.
> > > > > > > > >> > >>>>>> Do
> > > > > > > > >> > >>>>>>>> not
> > > > > > > > >> > >>>>>>>>>>> click links or open attachments unless you
> can
> > > > > confirm
> > > > > > > the
> > > > > > > > >> > >>>>> sender
> > > > > > > > >> > >>>>>>> and
> > > > > > > > >> > >>>>>>>>> know
> > > > > > > > >> > >>>>>>>>>>> the content is safe.
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique
> > provient
> > > > > dâun
> > > > > > > > >> > >>>>> expéditeur
> > > > > > > > >> > >>>>>>>>> externe.
> > > > > > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune
> > > piĂšce
> > > > > > > jointe
> > > > > > > > >> > >>> si
> > > > > > > > >> > >>>>> vous
> > > > > > > > >> > >>>>>>> ne
> > > > > > > > >> > >>>>>>>>> pouvez
> > > > > > > > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et
> si
> > > > vous
> > > > > > > nâĂȘtes
> > > > > > > > >> > >>> pas
> > > > > > > > >> > >>>>>>>> certain
> > > > > > > > >> > >>>>>>>>> que
> > > > > > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> From Guido's post:
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have
> > > been
> > > > > > > > renamed".
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> Naming is hard.
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> J.
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > > > > > > >> > >>>>> droiddev510e@gmail.com
> > > > > > > > >> > >>>>>>>>> <mailto:
> > > > > > > > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments,
> > dag
> > > is
> > > > > an
> > > > > > > > >> > >>> Airflow
> > > > > > > > >> > >>>>>> term
> > > > > > > > >> > >>>>>>>> now
> > > > > > > > >> > >>>>>>>>>>>> rather
> > > > > > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph".
> Believe
> > > it
> > > > or
> > > > > > > not,
> > > > > > > > >> > >>> I've
> > > > > > > > >> > >>>>>>>> worked
> > > > > > > > >> > >>>>>>>>> with
> > > > > > > > >> > >>>>>>>>>>>> work
> > > > > > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for
> > > years
> > > > > now
> > > > > > > > >> > >>> and are
> > > > > > > > >> > >>>>>> pro
> > > > > > > > >> > >>>>>>>>> devs
> > > > > > > > >> > >>>>>>>>>>> but
> > > > > > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a
> > > DAG!
> > > > > For
> > > > > > > > >> > >>> them,
> > > > > > > > >> > >>>>> dag
> > > > > > > > >> > >>>>>> =
> > > > > > > > >> > >>>>>>>>> Airflow.
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better
> > > terms
> > > > > but
> > > > > > > > >> > >>> will be
> > > > > > > > >> > >>>>> a
> > > > > > > > >> > >>>>>>>> major
> > > > > > > > >> > >>>>>>>>>>>> behavior
> > > > > > > > >> > >>>>>>>>>>>> change for the developer community, so
> we'll
> > > > need a
> > > > > > > solid
> > > > > > > > >> > >>>>> plan
> > > > > > > > >> > >>>>>> on
> > > > > > > > >> > >>>>>>>>> how to
> > > > > > > > >> > >>>>>>>>>>>> introduce it, when we do.
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share
> > about
> > > > the
> > > > > > > Great
> > > > > > > > >> > >>>>>> Renaming
> > > > > > > > >> > >>>>>>>> in
> > > > > > > > >> > >>>>>>>>> core
> > > > > > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We
> > could
> > > > > > probably
> > > > > > > > >> > >>> get
> > > > > > > > >> > >>>>>> some
> > > > > > > > >> > >>>>>>>>>>> learnings
> > > > > > > > >> > >>>>>>>>>>>> from there? Who knows!
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> Guido's blog (2009):
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > > >> > >>>>>>>>>>> <
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>
> > > > > > > >
> > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > > > > >> > >>>>>>> <
> > > > > > > > >> > >>>>>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > >
> https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > > > > >
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> Cheers,
> > > > > > > > >> > >>>>>>>>>>>> Omkar
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > > > > > > > >> > >>> weilee.rx@gmail.com
> > > > > > > > >> > >>>>>>>>> <mailto:
> > > > > > > > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>> I think we should probably just accept it
> as
> > > an
> > > > > > > airflow
> > > > > > > > >> > >>>>> term.
> > > > > > > > >> > >>>>>>> At
> > > > > > > > >> > >>>>>>>>> least,
> > > > > > > > >> > >>>>>>>>>>>>> thatâs how I understood it when I first
> used
> > > > > > Airflow.
> > > > > > > I
> > > > > > > > >> > >>>>> feel
> > > > > > > > >> > >>>>>>>>> renaming
> > > > > > > > >> > >>>>>>>>>>> it
> > > > > > > > >> > >>>>>>>>>>>> at
> > > > > > > > >> > >>>>>>>>>>>>> this stage would require considerable
> effort
> > > > from
> > > > > > > > >> > >>>>> maintainers
> > > > > > > > >> > >>>>>>> and
> > > > > > > > >> > >>>>>>>>>>>> existing
> > > > > > > > >> > >>>>>>>>>>>>> users without providing equivalent
> benefits.
> > > > > > > > >> > >>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>> Best,
> > > > > > > > >> > >>>>>>>>>>>>> Wei
> > > > > > > > >> > >>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > > > > > > >> > >>>>>> kaxilnaik@gmail.com
> > > > > > > > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe
> we
> > > > > > re-kindle
> > > > > > > > >> > >>>>> this
> > > > > > > > >> > >>>>>>>>> discussion
> > > > > > > > >> > >>>>>>>>>>>> for
> > > > > > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will
> cause
> > > too
> > > > > > much
> > > > > > > > >> > >>>>>>> disruption
> > > > > > > > >> > >>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance
> > > > Martineau
> > > > > > > > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > > > > > > > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > > > > > > > >> > >>>>>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those
> with
> > > > > Airflow
> > > > > > > > >> > >>>>>> experience
> > > > > > > > >> > >>>>>>>>> what a
> > > > > > > > >> > >>>>>>>>>>>> dag
> > > > > > > > >> > >>>>>>>>>>>>> is,
> > > > > > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow
> > > > attributes
> > > > > -
> > > > > > > > >> > >>> stuff
> > > > > > > > >> > >>>>>> like
> > > > > > > > >> > >>>>>>>>> dags
> > > > > > > > >> > >>>>>>>>>>>> being
> > > > > > > > >> > >>>>>>>>>>>>> a
> > > > > > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners.
> The
> > > > > > structure
> > > > > > > > >> > >>>>>> doesn't
> > > > > > > > >> > >>>>>>>>> come up.
> > > > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote
> is
> > > to
> > > > > > > > >> > >>> embrace
> > > > > > > > >> > >>>>> the
> > > > > > > > >> > >>>>>>> name
> > > > > > > > >> > >>>>>>>>> and
> > > > > > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure
> > > > aspect.
> > > > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram
> > Koka
> > > > > > > > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > > > > > > > >> > >>>>> vikram@astronomer.io.inva
> > > > > > > > >> > >>>>>>>>> lid>
> > > > > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I
> > > remember
> > > > > > > > >> > >>>>> struggling
> > > > > > > > >> > >>>>>>> with
> > > > > > > > >> > >>>>>>>>> this
> > > > > > > > >> > >>>>>>>>>>>>> when I
> > > > > > > > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > > > > > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of
> > it
> > > > > being
> > > > > > an
> > > > > > > > >> > >>>>>>>> established
> > > > > > > > >> > >>>>>>>>>>>> concept
> > > > > > > > >> > >>>>>>>>>>>>>>> now
> > > > > > > > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the
> > > > perspective
> > > > > > > > >> > >>> best
> > > > > > > > >> > >>>>>>>> expressed
> > > > > > > > >> > >>>>>>>>> by
> > > > > > > > >> > >>>>>>>>>>>>> Daniel
> > > > > > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a
> > word,
> > > > > rather
> > > > > > > > >> > >>> than
> > > > > > > > >> > >>>>>> the
> > > > > > > > >> > >>>>>>>>> computer
> > > > > > > > >> > >>>>>>>>>>>>>>> science
> > > > > > > > >> > >>>>>>>>>>>>>>>> concept.
> > > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>> Best regards,
> > > > > > > > >> > >>>>>>>>>>>>>>>> Vikram
> > > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM
> Oliveira,
> > > > Niko
> > > > > > > > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > > > > > > > >> > >>>>>> onikolas@amazon.com.inva
> > > > > > > > >> > >>>>>>>>> lid
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of:
> > > > You're
> > > > > > > > >> > >>> right
> > > > > > > > >> > >>>>>> Ryan,
> > > > > > > > >> > >>>>>>>> DAG
> > > > > > > > >> > >>>>>>>>>>> isn't
> > > > > > > > >> > >>>>>>>>>>>>>>> great
> > > > > > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing
> it
> > > > will
> > > > > > > > >> > >>> cause
> > > > > > > > >> > >>>>> much
> > > > > > > > >> > >>>>>>>> more
> > > > > > > > >> > >>>>>>>>>>>> wreckage
> > > > > > > > >> > >>>>>>>>>>>>>>>> than
> > > > > > > > >> > >>>>>>>>>>>>>>>>> it solves.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move
> > away
> > > > > from
> > > > > > > > >> > >>>>> defining
> > > > > > > > >> > >>>>>>>> DAG.
> > > > > > > > >> > >>>>>>>>> I
> > > > > > > > >> > >>>>>>>>>>>> think
> > > > > > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a
> > > > community
> > > > > > > > >> > >>> for a
> > > > > > > > >> > >>>>>> while
> > > > > > > > >> > >>>>>>>> now
> > > > > > > > >> > >>>>>>>>>>>> anyway,
> > > > > > > > >> > >>>>>>>>>>>>>>> so
> > > > > > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Niko
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> ________________________________
> > > > > > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <
> ash@apache.org
> > > > > > <mailto:
> > > > > > > > >> > >>>>>>>>> ash@apache.org>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024
> 9:06:39
> > AM
> > > > > > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > > > > >> > >>>>>> dev@airflow.apache.org
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should
> > > deprecate
> > > > > the
> > > > > > > > >> > >>> term
> > > > > > > > >> > >>>>>> "DAG"
> > > > > > > > >> > >>>>>>>>> for end
> > > > > > > > >> > >>>>>>>>>>>>>>> users
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from
> > > outside
> > > > of
> > > > > > the
> > > > > > > > >> > >>>>>>>>> organization.
> > > > > > > > >> > >>>>>>>>>>> Do
> > > > > > > > >> > >>>>>>>>>>>>> not
> > > > > > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless
> > you
> > > > can
> > > > > > > > >> > >>> confirm
> > > > > > > > >> > >>>>>> the
> > > > > > > > >> > >>>>>>>>> sender
> > > > > > > > >> > >>>>>>>>>>>> and
> > > > > > > > >> > >>>>>>>>>>>>>>>> know
> > > > > > > > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier
> électronique
> > > > > provient
> > > > > > > > >> > >>> dâun
> > > > > > > > >> > >>>>>>>>> expéditeur
> > > > > > > > >> > >>>>>>>>>>>>>>> externe.
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez
> > > aucune
> > > > > > piĂšce
> > > > > > > > >> > >>>>>> jointe
> > > > > > > > >> > >>>>>>> si
> > > > > > > > >> > >>>>>>>>> vous
> > > > > > > > >> > >>>>>>>>>>> ne
> > > > > > > > >> > >>>>>>>>>>>>>>>> pouvez
> > > > > > > > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de
> lâexpĂ©diteur
> > > et
> > > > si
> > > > > > > > >> > >>> vous
> > > > > > > > >> > >>>>>> nâĂȘtes
> > > > > > > > >> > >>>>>>>> pas
> > > > > > > > >> > >>>>>>>>>>>> certain
> > > > > > > > >> > >>>>>>>>>>>>>>>> que
> > > > > > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping
> > âdagsâ
> > > > as a
> > > > > > > > >> > >>> term â
> > > > > > > > >> > >>>>>>>> getting
> > > > > > > > >> > >>>>>>>>> to
> > > > > > > > >> > >>>>>>>>>>>>> re-use
> > > > > > > > >> > >>>>>>>>>>>>>>>>> puns like
> > > https://i.imgflip.com/1xhtwh.jpg
> > > > <
> > > > > > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind
> either
> > > way,
> > > > > > both
> > > > > > > > >> > >>>>> sides
> > > > > > > > >> > >>>>>>> have
> > > > > > > > >> > >>>>>>>>> good
> > > > > > > > >> > >>>>>>>>>>>>>>> reasons
> > > > > > > > >> > >>>>>>>>>>>>>>>>> presented.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>> -a
> > > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel
> > Standish
> > > > > > > > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA
> > > > <mailto:
> > > > > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID>
> > wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the
> > name
> > > > > comes
> > > > > > > > >> > >>> from,
> > > > > > > > >> > >>>>>>> "well,
> > > > > > > > >> > >>>>>>>>> no
> > > > > > > > >> > >>>>>>>>>>> one
> > > > > > > > >> > >>>>>>>>>>>>>>>>> actually
> > > > > > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something
> > up.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek
> > > > Potiuk
> > > > > <
> > > > > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic
> > > graph"
> > > > > is
> > > > > > > > >> > >>> the
> > > > > > > > >> > >>>>>>>>>>> tongue-twister,
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM
> Jarek
> > > > > Potiuk <
> > > > > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent
> > > wrote.
> > > > I
> > > > > > > > >> > >>> would
> > > > > > > > >> > >>>>> very
> > > > > > > > >> > >>>>>>>> much
> > > > > > > > >> > >>>>>>>>> want
> > > > > > > > >> > >>>>>>>>>>>> to
> > > > > > > > >> > >>>>>>>>>>>>>>> be
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> able
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without
> explaining
> > it
> > > > > > > > >> > >>> further.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG"
> at
> > a
> > > > talk
> > > > > > > > >> > >>> it's a
> > > > > > > > >> > >>>>>>>>>>>> tongue-twister,
> > > > > > > > >> > >>>>>>>>>>>>>>>> and
> > > > > > > > >> > >>>>>>>>>>>>>>>>> I
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall
> > how
> > > to
> > > > > > > > >> > >>> pronounce
> > > > > > > > >> > >>>>>> it
> > > > > > > > >> > >>>>>>>>>>> properly.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> J.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM
> Brent
> > > > > Bovenzi
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > > > > > > > >> > >>>>>>>> brent@astronomer.io.inva
> > > > > > > > >> > >>>>>>>>>> lid>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming
> > "DAG"
> > > > when
> > > > > > > > >> > >>>>> starting
> > > > > > > > >> > >>>>>> on
> > > > > > > > >> > >>>>>>>>> AIP-38
> > > > > > > > >> > >>>>>>>>>>> to
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> modernize
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or
> > "workflow"
> > > > are
> > > > > > more
> > > > > > > > >> > >>>>>>>>> descriptive of
> > > > > > > > >> > >>>>>>>>>>>> what
> > > > > > > > >> > >>>>>>>>>>>>>>>> one
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> is
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed
> > Acyclic
> > > > > Graph
> > > > > > > > >> > >>> is an
> > > > > > > > >> > >>>>>>>>>>> implementation
> > > > > > > > >> > >>>>>>>>>>>>>>>>> detail.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish,
> at
> > > > this
> > > > > > > > >> > >>> point
> > > > > > > > >> > >>>>>> "DAG"
> > > > > > > > >> > >>>>>>>> has
> > > > > > > > >> > >>>>>>>>>>> become
> > > > > > > > >> > >>>>>>>>>>>>>>>> "dag"
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> , a
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in
> > > > community
> > > > > > > > >> > >>>>>> discussion,
> > > > > > > > >> > >>>>>>>>> Airflow
> > > > > > > > >> > >>>>>>>>>>>>>>>> Summit
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in
> the
> > > UI.
> > > > > > Let's
> > > > > > > > >> > >>>>>> embrace
> > > > > > > > >> > >>>>>>>>> "dag".
> > > > > > > > >> > >>>>>>>>>>> A
> > > > > > > > >> > >>>>>>>>>>>>>>> user
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>> just
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the
> > > > > technical
> > > > > > > > >> > >>>>> concept
> > > > > > > > >> > >>>>>>>>> behind
> > > > > > > > >> > >>>>>>>>>>> that
> > > > > > > > >> > >>>>>>>>>>>>>>>>> word. I
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort
> than
> > > > > > > > >> > >>> refactoring so
> > > > > > > > >> > >>>>>>> much
> > > > > > > > >> > >>>>>>>>> code,
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack
> > > > overflow
> > > > > > > > >> > >>>>> questions,
> > > > > > > > >> > >>>>>>>> etc.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM
> > > Daniel
> > > > > > > > >> > >>> Standish
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> <daniel.standish@astronomer.io.inva
> > > > > > <mailto:
> > > > > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid>
> > wrote:
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like
> > > introducing
> > > > a
> > > > > > lot
> > > > > > > > >> > >>> of
> > > > > > > > >> > >>>>>> pain
> > > > > > > > >> > >>>>>>>> for
> > > > > > > > >> > >>>>>>>>>>>>>>>>> questionable
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def
> sympathetic
> > to
> > > > the
> > > > > > > > >> > >>> idea. I
> > > > > > > > >> > >>>>>>> agree
> > > > > > > > >> > >>>>>>>>> the
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> association
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is
> > not
> > > > > > helpful.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer
> here
> > > > some
> > > > > > less
> > > > > > > > >> > >>>>>>> invasive
> > > > > > > > >> > >>>>>>>>>>>>>>> mitigations.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter
> what
> > is
> > > > to
> > > > > > > > >> > >>>>>> de-emphasize
> > > > > > > > >> > >>>>>>>> the
> > > > > > > > >> > >>>>>>>>>>> math
> > > > > > > > >> > >>>>>>>>>>>>>>> nerd
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in
> > docs
> > > /
> > > > > > > > >> > >>> website /
> > > > > > > > >> > >>>>>> etc,
> > > > > > > > >> > >>>>>>>>> *never
> > > > > > > > >> > >>>>>>>>>>>>>>>> define*
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a
> > directed
> > > > > > acyclic
> > > > > > > > >> > >>>>>> graph.
> > > > > > > > >> > >>>>>>>>> Always
> > > > > > > > >> > >>>>>>>>>>>>>>> define
> > > > > > > > >> > >>>>>>>>>>>>>>>>> it
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks,
> > > workflow
> > > > > > etc.
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part
> > of
> > > it
> > > > > is
> > > > > > > > >> > >>> like a
> > > > > > > > >> > >>>>>>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Oh no.. Swords, swords, .... :)
Hello Brian ! Nice to see you back in the conversations. We've been
discussing things like this before a few times even over the phone, I
recall.
Yes, I think it's partially marketing, I agree. And marketing that is
important for the project, which we as engineers, should also at least
understand. We should worry about how our work is perceived by
non-engineers or at least not-so-much-deep-into-technical-correctness
engineers who just want to get stuff done. This is at least my goal of
"communicating" such stuff through documentation - recognise that the vast
majority of our users just want to get things done and don't really care
about abstract mathematical concepts, they might not even be aware of. Our
role in this project is not to teach people math or algorithmics, but to
help them do their job. That's our primary goal.
And to that - more importantly (for the reasons i and Constance explained)
we **might** soon not have "technically Directed Acyclic Graphs" - we do
not know if we go that route, but IMHO, it's likely. Then from a good
engineering point of view, figuring a better name that would ease the
transition is not only smart, but also fair, and it will make our life
easier, I am not a fan of being "100% precise and correct" if it makes my
future life harder. As an experienced engineer I am happy to go down the
path of making definitions more blurry, if it is still "good definition" -
fitting the purpose, and making it easier to transition to the place where
the old definition will no longer be true.
That's what I consider being a great engineer is - to not only focus on
minute details and definition correctness, but also understanding of my
users, communicating with them in the way that makes it easier for them to
relate to the project and "do their job", and to foresee and clear the path
for future evolution of my product.
I hope this explanation can provide a bit more context - at least how I
understand it, and I hope you will see it's not a "because we want it"
change. It has some real grounding, reasoning, expectations and is
something that few of us have been thinking about for a while. So please do
not see it as "marketing driven" change. For me it's "engineering driven
change that takes into account marketing".
J.
On Tue, Feb 18, 2025 at 7:20âŻPM Brian Greene <
brian@heisenbergwoodworking.com> wrote:
> I disagree with renaming it, or removing the logical idea of directed
> acyclic graph, or redefining the acronym. Itâs less âmathâ to me, and more
> âdata structureâ, and a lot of things work the way they do because a dag IS
> a DAG.
>
> Whatâs the problem with calling something EXACTLY what it is? So rarely
> does a technical team get it so right⊠this feels like a discussion with
> marketing people, wanting to obfuscate something completely reasonable, for
> no functional value.
>
> How much energy can be spent making this scenario more common? âwhatâs DAG
> vs Dag vs Oh you mean DaaG :-(⊠or whateverâ. And why would one want to?
>
> Brian
>
> On Tue, Feb 18, 2025 at 11:45âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > -0.5
> > > Even though the new UI prefers "Dag", I don't think that means "DAG"
> is
> > incorrect.
> >
> > I think we have a bunch of related things discussed here - I am not
> exactly
> > sure what the -0.5 in this context means Brent. Would it be possible to
> > clarify ?
> >
> > On Tue, Feb 18, 2025 at 6:01âŻPM Ambika Garg <am...@gmail.com>
> > wrote:
> >
> > > +1 would vote for
> > > Airflow's *D*ynamic *A*ction *G*raph (DAG)
> > >
> > > *Dynamic*: Reflects the capability of Airflow to handle workflows that
> > > arenât strictly acyclic, but can adapt to changes, iterations and loops
> > > within broader context of workflow.
> > >
> > > *Action*: Emphasizes that airflow is fundamentally about orchestrating
> > > actions or tasks. Whether its data processing, model training or any
> > other
> > > form of computation. It also conveys operational aspect of what Airflow
> > > does.
> > >
> > > *Graph*: Maintains the core concept that workflows in Airflow are
> > > visualized and managed as graphs. This term keeps the idea of
> > dependencies
> > > and relationships between tasks intact.
> > >
> > > Ambika.
> > >
> > > On Tue, Feb 18, 2025 at 11:46âŻAM Brent Bovenzi
> > <brent@astronomer.io.invalid
> > > >
> > > wrote:
> > >
> > > > -0.5
> > > >
> > > > Even though the new UI prefers "Dag", I don't think that means "DAG"
> is
> > > > incorrect.
> > > >
> > > > On Tue, Feb 18, 2025 at 11:36âŻAM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >
> > > > > I love "Dynamic Airflow Graph" +10 to that Constance.
> > > > >
> > > > > And yes. I fully agree with your "somewhat" acyclic statement. I
> saw
> > > > people
> > > > > mentioning "event-driven" workflows to better describe the AI/ML,
> but
> > > > that
> > > > > for me is a synonym of "completely unmanageable workflows" (similar
> > to
> > > > > micro-services) - what I was alluding to was exactly this -
> something
> > > > like
> > > > > curren DAG with parts of it being potentially cyclic - so we are
> > > > completely
> > > > > aligned here :). And that's what make it "not DAG - in mathematical
> > > > sense"
> > > > > - even more reason to depart from it,
> > > > >
> > > > > J.
> > > > >
> > > > > On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
> > > > > <co...@astronomer.io.invalid> wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I appreciate the discussion about how we position the concept of
> a
> > > DAG
> > > > in
> > > > > > Airflow. While I agree with updating documentation to
> de-emphasize
> > > the
> > > > > > strict mathematical definition, I do not think we should change
> the
> > > > > public
> > > > > > interface to use "dag" instead of "DAG" everywhere. It adds
> > > > > > unnecessary churn and breaks consistency across the ecosystem.
> > > > > >
> > > > > > Regarding the broader discussion on whether DAGs in Airflow
> should
> > > > remain
> > > > > > acyclic: I acknowledge that some workflows, particularly in ML
> and
> > > AI,
> > > > > > require cyclical behaviour (example: iterative refinement until a
> > > > > condition
> > > > > > is met). However, I think it's important to recognize that even
> in
> > > > those
> > > > > > cases, the workflow itself still has an overall start and end.
> It's
> > > not
> > > > > an
> > > > > > unbounded loop but rather a controlled execution process that
> > happens
> > > > to
> > > > > > have cycles *within* it.
> > > > > >
> > > > > > That being said, I do see value in reinterpreting "DAG" to better
> > > align
> > > > > > with how Airflow is actually used, and love shifting away from
> > > > "Directed
> > > > > > Acyclic Graph" in favour of something more intuitive. However
> "Data
> > > > > Airflow
> > > > > > Graph" makes it sound like we're defining Airflow itself rather
> > than
> > > > > > describing the structure of workflows within it.
> > > > > >
> > > > > > If we want a new meaning for DAG that resonates with our base
> while
> > > > > keeping
> > > > > > Airflow's identity clear, I am proposing three alternatives:
> > > > > >
> > > > > >
> > > > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while
> > > > emphasizing
> > > > > > flexibility
> > > > > > 2. *Distributed Automation Graph*: Highlights execution across
> > > > systems
> > > > > > and automations without tying it strictly to data
> > > > > > 3. *Directed Activity Graph*: Retains the useful meaning of
> > > > "Directed"
> > > > > > while shifting away from strict acyclic constraints. (My
> > personal
> > > > > > favourite)
> > > > > >
> > > > > > I think any shift in terminology should be driven by what best
> > > > > communicates
> > > > > > Airflow's role workflow automation, rather than just distancing
> > > > ourselves
> > > > > > from the mathematical definition.
> > > > > >
> > > > > > Curious to hear more thoughts!
> > > > > >
> > > > > > On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > > > >
> > > > > > > Sorry for the few typos ... My slight dyslexia did not help and
> > > > > > autocorrect
> > > > > > > did not help either this time : dept -> debt, routed -> rooted
> > and
> > > a
> > > > > few
> > > > > > > other small typos :) .
> > > > > > >
> > > > > > > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <jarek@potiuk.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > First of all - yes, agree with Lee and Ash, I think we do not
> > > need
> > > > to
> > > > > > > > change the public interface, I'd also be -0 on that, but
> docs,
> > > > > internal
> > > > > > > > strings, yes we could.
> > > > > > > >
> > > > > > > > But .... After thinking a bit and looking at the discussion
> > here,
> > > > > where
> > > > > > > > people have concerns, I have an alternative take. And I know
> > this
> > > > > might
> > > > > > > be
> > > > > > > > controversial, and some people might find it confusing, but
> why
> > > > don't
> > > > > > we
> > > > > > > > come up with a new acronym for DAG, one that will be more
> > airflow
> > > > and
> > > > > > > less
> > > > > > > > "mathematics/algorithm". And I have, I think a good reason
> for
> > > it,
> > > > > and
> > > > > > I
> > > > > > > > even have a proposal. Bear with me,
> > > > > > > >
> > > > > > > > DAG => *Data Airflow Graph*
> > > > > > > >
> > > > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1
> > now
> > > > > (with
> > > > > > > > exception of some niche products - you know which one I speak
> > > > about -
> > > > > > > which
> > > > > > > > are unfortunate enough to use the "Dag" in their name - I
> will
> > > > > explain
> > > > > > > why
> > > > > > > > this is unfortunate, IMHO).
> > > > > > > >
> > > > > > > > We are lucky to have Airflow for "A" in the acronym and we
> can
> > > make
> > > > > it
> > > > > > > > even more "synonymous" -> Airflow <> DAG with (A) being
> > Airflow.
> > > > > > > >
> > > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > > > > > > >
> > > > > > > > Not only because "Directed" and "Acyclic" are mostly not in
> the
> > > > > > > vocabulary
> > > > > > > > of most of our users - as Ryan, starting this thread
> explained
> > > > yes, I
> > > > > > > agree
> > > > > > > > with it, but also I think there is a more important reason -
> I
> > > > > simply
> > > > > > > > believe in the future our DAGs might not (and likely will
> not)
> > be
> > > > > > > "acyclic".
> > > > > > > >
> > > > > > > > I know for some it might be herezy and we have not discussed
> > any
> > > of
> > > > > > that
> > > > > > > > yet, but more and more I keep on hearing that Machine
> Learning
> > > > > > Workflows
> > > > > > > of
> > > > > > > > the future are often somewhat cyclic. In our case there is
> one
> > > type
> > > > > of
> > > > > > > > repeat-in-cycle behaviour - we have "task retry on failure" -
> > but
> > > > > this
> > > > > > is
> > > > > > > > only on failure and it basically mens "idempotent" retry - we
> > > want
> > > > to
> > > > > > > redo
> > > > > > > > something and get single result out of that and basically
> > discard
> > > > the
> > > > > > > > previous result. But machine learning/AI workflows are
> > > different. I
> > > > > > had a
> > > > > > > > lot of conversations recently (I even co-organized "AI
> > low-level
> > > > > > > > engineering and hacking" devroom to learn more about those) -
> > > and I
> > > > > > keep
> > > > > > > on
> > > > > > > > hearing that "Repeat a sequence of things several times and
> > > refine"
> > > > > is
> > > > > > > > precisely what AI / ML workflows will need - especially with
> > > > Agentic
> > > > > > > > workflows - often we have cooperating agents which **could**
> be
> > > > part
> > > > > of
> > > > > > > the
> > > > > > > > same DAG, but currently in Airflow where our DAGs are
> > "Acyclic"
> > > we
> > > > > can
> > > > > > > > only express "repeat several time and refine" as a single
> task
> > > that
> > > > > > will
> > > > > > > do
> > > > > > > > such refining. If we would like to split such an "agent"
> doing
> > > > their
> > > > > > own
> > > > > > > > work in "cycles" and communicating with other "agents" doing
> > the
> > > > > same -
> > > > > > > > represented as a multi-task-entity, we have no way to express
> > it
> > > > > using
> > > > > > > our
> > > > > > > > current "acyclic" semantics.
> > > > > > > >
> > > > > > > > But - other than our implementation makes that assumption
> > > > currently -
> > > > > > > > there is nothing fundamental with Airflow being able to run
> > such
> > > > > > > "partially
> > > > > > > > cyclic workflows". We have all the pieces, we would have to
> > > change
> > > > > > > > dependency calculation to allow for cycles, update UI to
> handle
> > > > > cycles
> > > > > > > > representation - and we could in theory quite seamlessly,
> > support
> > > > > > > "cyclic"
> > > > > > > > workflows.
> > > > > > > >
> > > > > > > > I know, I know, that's the first time we hear it and that's
> > huge
> > > > > > change -
> > > > > > > > of course, I do not want to start discussing any details of
> it,
> > > not
> > > > > > > before
> > > > > > > > Airflow 3.0 is out - but I think we should be open that at
> some
> > > > point
> > > > > > of
> > > > > > > > time the "Acyclic" property is going to go away, And my
> > personal
> > > > > > > intuition
> > > > > > > > tells me that a year from now we will have acyclic workflows
> > > > > (assuming
> > > > > > > that
> > > > > > > > all the foundational work we do on Airflow 3 will really help
> > in
> > > > > > > decreasing
> > > > > > > > all the technical dept, and will open our minds to new
> ideas).
> > > > > > > >
> > > > > > > > That's why I think - the sooner we get rid of "Directed
> > Acyclic"
> > > -
> > > > > the
> > > > > > > > better. that will - even for us, maintainer free us from
> being
> > > > routed
> > > > > > in
> > > > > > > > the acyclic nature of Airflow.
> > > > > > > >
> > > > > > > > How about we change to "Data Airflow Graph" (or any other
> good
> > > > > acronym
> > > > > > > > which has (A)irflow instead of (A)cyclic). The name is just a
> > > > > proposal.
> > > > > > > >
> > > > > > > > Now.. I go hide somewhere as I believe that might cause a
> > > > "flamewar"
> > > > > > :),
> > > > > > > > unless on request of Ryan you already sheathed your swords in
> > the
> > > > > > > > discussion (which seems everyone did).
> > > > > > > >
> > > > > > > > But honestly - I would really love what you think of "let's
> > just
> > > > > break
> > > > > > > > with Acyclic NOW" and shape our future by using a different
> > > > acronym.
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > > > > > > ephraimanierobi@gmail.com> wrote:
> > > > > > > >
> > > > > > > >> When I started contributing to Airflow, I had to read up on
> > DAG,
> > > > and
> > > > > > > now I
> > > > > > > >> know what it means. I wonder if we are about to have users
> who
> > > use
> > > > > the
> > > > > > > >> term
> > > > > > > >> Dag without knowing that there's a DAG.
> > > > > > > >>
> > > > > > > >> Slightly concerned.
> > > > > > > >>
> > > > > > > >> -ephraim
> > > > > > > >>
> > > > > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com>
> > > > wrote:
> > > > > > > >>
> > > > > > > >> > For user-facing things (e.g., `from airflow import DAG`),
> > Iâm
> > > > more
> > > > > > > like
> > > > > > > >> > -0. But for documentation, docstring, internal things and
> > > etc.,
> > > > we
> > > > > > > >> probably
> > > > > > > >> > could still change most of them?
> > > > > > > >> >
> > > > > > > >> > Best,
> > > > > > > >> > Wei
> > > > > > > >> >
> > > > > > > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <
> > > > ash@apache.org>
> > > > > > > >> wrote:
> > > > > > > >> > >
> > > > > > > >> > > I also donât personally think itâs worth the pain (not
> to
> > > > > mention
> > > > > > > >> > backcompat workaround) to rename DAG to Dag, so Iâd be
> -0.5
> > on
> > > > > that.
> > > > > > > >> > >
> > > > > > > >> > > -ash
> > > > > > > >> > >
> > > > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > > > > > > daniel.imberman@gmail.com
> > > > > > > >> >
> > > > > > > >> > wrote:
> > > > > > > >> > >>
> > > > > > > >> > >> I think my biggest concern is a marketing one and not a
> > > > > technical
> > > > > > > >> one.
> > > > > > > >> > >>
> > > > > > > >> > >> As has been mentioned on the thread the terms airflow
> and
> > > dag
> > > > > are
> > > > > > > >> kind
> > > > > > > >> > of
> > > > > > > >> > >> synonymous and I certainly donât want to give the
> > > impression
> > > > > that
> > > > > > > we
> > > > > > > >> are
> > > > > > > >> > >> breaking more than we are breaking.
> > > > > > > >> > >>
> > > > > > > >> > >> I wouldnât die on this hill, but Iâm slightly
> concerned.
> > > > > > > >> > >>
> > > > > > > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <
> > weilee@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >> > >>
> > > > > > > >> > >>> Iâm not sure about adding ruff rules here đ€ I think
> > ruff
> > > > > rules
> > > > > > > are
> > > > > > > >> > best
> > > > > > > >> > >>> suited for user-facing things but not the airflow code
> > > base
> > > > > > > itself.
> > > > > > > >> If
> > > > > > > >> > what
> > > > > > > >> > >>> we mean is adding a rule to avoid users using "DAG"
> > > *after*
> > > > we
> > > > > > > >> rename
> > > > > > > >> > it,
> > > > > > > >> > >>> it's definitely a +1000.
> > > > > > > >> > >>>
> > > > > > > >> > >>> I just created GitHub issues for this removing "DAG"
> > > > > > > >> > >>>
> > > > > > > >> > >>> https://github.com/apache/airflow/issues/46842
> > > > > > > >> > >>> https://github.com/apache/airflow/issues/46843
> > > > > > > >> > >>>
> > > > > > > >> > >>> One thing I'm not sure about is whether we want to get
> > rid
> > > > of
> > > > > > > `from
> > > > > > > >> > >>> airflow import DAG` as well. đ€
> > > > > > > >> > >>>
> > > > > > > >> > >>> Best,
> > > > > > > >> > >>> Wei
> > > > > > > >> > >>>
> > > > > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > > > > > > >> > >>>> Hard to say until it's looked at :)
> > > > > > > >> > >>>>
> > > > > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > > > > > > >> aritrabasu1999@gmail.com
> > > > > > > >> > >
> > > > > > > >> > >>>> wrote:
> > > > > > > >> > >>>>
> > > > > > > >> > >>>>> I can take it up, it's mostly just a doc update
> right?
> > > Or
> > > > > are
> > > > > > we
> > > > > > > >> > doing
> > > > > > > >> > >>> code
> > > > > > > >> > >>>>> files replacement too?
> > > > > > > >> > >>>>> --
> > > > > > > >> > >>>>> Regards,
> > > > > > > >> > >>>>> Aritra Basu
> > > > > > > >> > >>>>>
> > > > > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <
> > > > > jarek@potiuk.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > >>>>>
> > > > > > > >> > >>>>>> Sounds like another +1000 files big PR is coming
> :) ?
> > > Any
> > > > > > > >> volunteers
> > > > > > > >> > >>> to
> > > > > > > >> > >>>>>> make it ? It's fun.
> > > > > > > >> > >>>>>>
> > > > > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > > > > > > droiddev510e@gmail.com>
> > > > > > > >> > >>> wrote:
> > > > > > > >> > >>>>>>
> > > > > > > >> > >>>>>>> +1 for ruff rules :)
> > > > > > > >> > >>>>>>>
> > > > > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace
> > 'DAG'
> > > > in
> > > > > > the
> > > > > > > >> > >>> Airflow
> > > > > > > >> > >>>>>>> docs, in line with the new UI changes and to make
> > the
> > > > > > renaming
> > > > > > > >> > >>>>>>> consistent across user-facing pages.
> > > > > > > >> > >>>>>>>
> > > > > > > >> > >>>>>>> Regards,
> > > > > > > >> > >>>>>>> Omkar
> > > > > > > >> > >>>>>>>
> > > > > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > > > > > > jarek@potiuk.com>
> > > > > > > >> > >>> wrote:
> > > > > > > >> > >>>>>>>
> > > > > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that
> :)
> > > > > > > >> > >>>>>>>>
> > > > > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <
> > > > > weilee@apache.org
> > > > > > >
> > > > > > > >> > >>> wrote:
> > > > > > > >> > >>>>>>>>
> > > > > > > >> > >>>>>>>>> It seems that our current conclusion is to use
> > "dag"
> > > > or
> > > > > > > "Dag"
> > > > > > > >> > >>>>> instead
> > > > > > > >> > >>>>>>> of
> > > > > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all
> > "DAG"
> > > > in
> > > > > > the
> > > > > > > >> > >>>>> codebase
> > > > > > > >> > >>>>>>> with
> > > > > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which
> > it
> > > > > might
> > > > > > be
> > > > > > > >> > >>> đ€),
> > > > > > > >> > >>>>>>> should
> > > > > > > >> > >>>>>>>> we
> > > > > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the
> > > following
> > > > > PRs?
> > > > > > > >> > >>>>>>>>>
> > > > > > > >> > >>>>>>>>> Best,
> > > > > > > >> > >>>>>>>>> Wei
> > > > > > > >> > >>>>>>>>>
> > > > > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new
> 3.0
> > > UI:
> > > > > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > > > > > > >> > >>>>>>>>>> Let me know if that's the direction we want to
> > go.
> > > > > > > >> > >>>>>>>>>>
> > > > > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo,
> > > Rajeshwar
> > > > > > > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > > > > > > >> > >>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best,
> "dag"
> > > is
> > > > > > > >> > >>> synonymous
> > > > > > > >> > >>>>>> with
> > > > > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk
> and
> > > > > explain
> > > > > > > >> > >>> what
> > > > > > > >> > >>>>>>> Airflow
> > > > > > > >> > >>>>>>>>> is
> > > > > > > >> > >>>>>>>>>>> all about. Although folks would ask, I donât
> > ever
> > > > use
> > > > > > the
> > > > > > > >> > >>>>>>>> mathematical
> > > > > > > >> > >>>>>>>>>>> definition of DAG.
> > > > > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing
> > for
> > > > many
> > > > > > > >> > >>> users
> > > > > > > >> > >>>>>> making
> > > > > > > >> > >>>>>>>>> such a
> > > > > > > >> > >>>>>>>>>>> change - I would rather direct that energy to
> > > > > appending
> > > > > > > the
> > > > > > > >> > >>>>>> Oxford
> > > > > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to
> > > > > workflows
> > > > > > in
> > > > > > > >> > >>>>>> Airflow.
> > > > > > > >> > >>>>>>>> __
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> -- Rajesh
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> > > > > > jarek@potiuk.com
> > > > > > > >> > >>>>>> <mailto:
> > > > > > > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of
> > the
> > > > > > > >> > >>>>> organization.
> > > > > > > >> > >>>>>> Do
> > > > > > > >> > >>>>>>>> not
> > > > > > > >> > >>>>>>>>>>> click links or open attachments unless you can
> > > > confirm
> > > > > > the
> > > > > > > >> > >>>>> sender
> > > > > > > >> > >>>>>>> and
> > > > > > > >> > >>>>>>>>> know
> > > > > > > >> > >>>>>>>>>>> the content is safe.
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique
> provient
> > > > dâun
> > > > > > > >> > >>>>> expéditeur
> > > > > > > >> > >>>>>>>>> externe.
> > > > > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune
> > piĂšce
> > > > > > jointe
> > > > > > > >> > >>> si
> > > > > > > >> > >>>>> vous
> > > > > > > >> > >>>>>>> ne
> > > > > > > >> > >>>>>>>>> pouvez
> > > > > > > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> > > vous
> > > > > > nâĂȘtes
> > > > > > > >> > >>> pas
> > > > > > > >> > >>>>>>>> certain
> > > > > > > >> > >>>>>>>>> que
> > > > > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> From Guido's post:
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have
> > been
> > > > > > > renamed".
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> Naming is hard.
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> J.
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > > > > > >> > >>>>> droiddev510e@gmail.com
> > > > > > > >> > >>>>>>>>> <mailto:
> > > > > > > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments,
> dag
> > is
> > > > an
> > > > > > > >> > >>> Airflow
> > > > > > > >> > >>>>>> term
> > > > > > > >> > >>>>>>>> now
> > > > > > > >> > >>>>>>>>>>>> rather
> > > > > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe
> > it
> > > or
> > > > > > not,
> > > > > > > >> > >>> I've
> > > > > > > >> > >>>>>>>> worked
> > > > > > > >> > >>>>>>>>> with
> > > > > > > >> > >>>>>>>>>>>> work
> > > > > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for
> > years
> > > > now
> > > > > > > >> > >>> and are
> > > > > > > >> > >>>>>> pro
> > > > > > > >> > >>>>>>>>> devs
> > > > > > > >> > >>>>>>>>>>> but
> > > > > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a
> > DAG!
> > > > For
> > > > > > > >> > >>> them,
> > > > > > > >> > >>>>> dag
> > > > > > > >> > >>>>>> =
> > > > > > > >> > >>>>>>>>> Airflow.
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better
> > terms
> > > > but
> > > > > > > >> > >>> will be
> > > > > > > >> > >>>>> a
> > > > > > > >> > >>>>>>>> major
> > > > > > > >> > >>>>>>>>>>>> behavior
> > > > > > > >> > >>>>>>>>>>>> change for the developer community, so we'll
> > > need a
> > > > > > solid
> > > > > > > >> > >>>>> plan
> > > > > > > >> > >>>>>> on
> > > > > > > >> > >>>>>>>>> how to
> > > > > > > >> > >>>>>>>>>>>> introduce it, when we do.
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share
> about
> > > the
> > > > > > Great
> > > > > > > >> > >>>>>> Renaming
> > > > > > > >> > >>>>>>>> in
> > > > > > > >> > >>>>>>>>> core
> > > > > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We
> could
> > > > > probably
> > > > > > > >> > >>> get
> > > > > > > >> > >>>>>> some
> > > > > > > >> > >>>>>>>>>>> learnings
> > > > > > > >> > >>>>>>>>>>>> from there? Who knows!
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> Guido's blog (2009):
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>
> > > > > > > >> > >>>>>>>
> > > > > > > >> > >>>>>
> > > > > > > >> > >>>
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > >> > >>>>>>>>>>> <
> > > > > > > >> > >>>>>>>>>
> > > > > > > >> > >>>>>>>
> > > > > > > >> > >>>>>
> > > > > > > >> > >>>
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > >> > >>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>
> > > > > > >
> https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > > > >> > >>>>>>> <
> > > > > > > >> > >>>>>>>>>>>
> > > > > > > >> > >>>>>
> > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > > > >
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> Cheers,
> > > > > > > >> > >>>>>>>>>>>> Omkar
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > > > > > > >> > >>> weilee.rx@gmail.com
> > > > > > > >> > >>>>>>>>> <mailto:
> > > > > > > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > > > > > > >> > >>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as
> > an
> > > > > > airflow
> > > > > > > >> > >>>>> term.
> > > > > > > >> > >>>>>>> At
> > > > > > > >> > >>>>>>>>> least,
> > > > > > > >> > >>>>>>>>>>>>> thatâs how I understood it when I first used
> > > > > Airflow.
> > > > > > I
> > > > > > > >> > >>>>> feel
> > > > > > > >> > >>>>>>>>> renaming
> > > > > > > >> > >>>>>>>>>>> it
> > > > > > > >> > >>>>>>>>>>>> at
> > > > > > > >> > >>>>>>>>>>>>> this stage would require considerable effort
> > > from
> > > > > > > >> > >>>>> maintainers
> > > > > > > >> > >>>>>>> and
> > > > > > > >> > >>>>>>>>>>>> existing
> > > > > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > > > > > > >> > >>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>> Best,
> > > > > > > >> > >>>>>>>>>>>>> Wei
> > > > > > > >> > >>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > > > > > >> > >>>>>> kaxilnaik@gmail.com
> > > > > > > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we
> > > > > re-kindle
> > > > > > > >> > >>>>> this
> > > > > > > >> > >>>>>>>>> discussion
> > > > > > > >> > >>>>>>>>>>>> for
> > > > > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause
> > too
> > > > > much
> > > > > > > >> > >>>>>>> disruption
> > > > > > > >> > >>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance
> > > Martineau
> > > > > > > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > > > > > > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > > > > > > >> > >>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with
> > > > Airflow
> > > > > > > >> > >>>>>> experience
> > > > > > > >> > >>>>>>>>> what a
> > > > > > > >> > >>>>>>>>>>>> dag
> > > > > > > >> > >>>>>>>>>>>>> is,
> > > > > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow
> > > attributes
> > > > -
> > > > > > > >> > >>> stuff
> > > > > > > >> > >>>>>> like
> > > > > > > >> > >>>>>>>>> dags
> > > > > > > >> > >>>>>>>>>>>> being
> > > > > > > >> > >>>>>>>>>>>>> a
> > > > > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The
> > > > > structure
> > > > > > > >> > >>>>>> doesn't
> > > > > > > >> > >>>>>>>>> come up.
> > > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is
> > to
> > > > > > > >> > >>> embrace
> > > > > > > >> > >>>>> the
> > > > > > > >> > >>>>>>> name
> > > > > > > >> > >>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure
> > > aspect.
> > > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram
> Koka
> > > > > > > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > > > > > > >> > >>>>> vikram@astronomer.io.inva
> > > > > > > >> > >>>>>>>>> lid>
> > > > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I
> > remember
> > > > > > > >> > >>>>> struggling
> > > > > > > >> > >>>>>>> with
> > > > > > > >> > >>>>>>>>> this
> > > > > > > >> > >>>>>>>>>>>>> when I
> > > > > > > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > > > > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of
> it
> > > > being
> > > > > an
> > > > > > > >> > >>>>>>>> established
> > > > > > > >> > >>>>>>>>>>>> concept
> > > > > > > >> > >>>>>>>>>>>>>>> now
> > > > > > > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the
> > > perspective
> > > > > > > >> > >>> best
> > > > > > > >> > >>>>>>>> expressed
> > > > > > > >> > >>>>>>>>> by
> > > > > > > >> > >>>>>>>>>>>>> Daniel
> > > > > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a
> word,
> > > > rather
> > > > > > > >> > >>> than
> > > > > > > >> > >>>>>> the
> > > > > > > >> > >>>>>>>>> computer
> > > > > > > >> > >>>>>>>>>>>>>>> science
> > > > > > > >> > >>>>>>>>>>>>>>>> concept.
> > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>> Best regards,
> > > > > > > >> > >>>>>>>>>>>>>>>> Vikram
> > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira,
> > > Niko
> > > > > > > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > > > > > > >> > >>>>>> onikolas@amazon.com.inva
> > > > > > > >> > >>>>>>>>> lid
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of:
> > > You're
> > > > > > > >> > >>> right
> > > > > > > >> > >>>>>> Ryan,
> > > > > > > >> > >>>>>>>> DAG
> > > > > > > >> > >>>>>>>>>>> isn't
> > > > > > > >> > >>>>>>>>>>>>>>> great
> > > > > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it
> > > will
> > > > > > > >> > >>> cause
> > > > > > > >> > >>>>> much
> > > > > > > >> > >>>>>>>> more
> > > > > > > >> > >>>>>>>>>>>> wreckage
> > > > > > > >> > >>>>>>>>>>>>>>>> than
> > > > > > > >> > >>>>>>>>>>>>>>>>> it solves.
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move
> away
> > > > from
> > > > > > > >> > >>>>> defining
> > > > > > > >> > >>>>>>>> DAG.
> > > > > > > >> > >>>>>>>>> I
> > > > > > > >> > >>>>>>>>>>>> think
> > > > > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a
> > > community
> > > > > > > >> > >>> for a
> > > > > > > >> > >>>>>> while
> > > > > > > >> > >>>>>>>> now
> > > > > > > >> > >>>>>>>>>>>> anyway,
> > > > > > > >> > >>>>>>>>>>>>>>> so
> > > > > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > > >> > >>>>>>>>>>>>>>>>> Niko
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> ________________________________
> > > > > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org
> > > > > <mailto:
> > > > > > > >> > >>>>>>>>> ash@apache.org>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39
> AM
> > > > > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > > > >> > >>>>>> dev@airflow.apache.org
> > > > > > > >> > >>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should
> > deprecate
> > > > the
> > > > > > > >> > >>> term
> > > > > > > >> > >>>>>> "DAG"
> > > > > > > >> > >>>>>>>>> for end
> > > > > > > >> > >>>>>>>>>>>>>>> users
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from
> > outside
> > > of
> > > > > the
> > > > > > > >> > >>>>>>>>> organization.
> > > > > > > >> > >>>>>>>>>>> Do
> > > > > > > >> > >>>>>>>>>>>>> not
> > > > > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless
> you
> > > can
> > > > > > > >> > >>> confirm
> > > > > > > >> > >>>>>> the
> > > > > > > >> > >>>>>>>>> sender
> > > > > > > >> > >>>>>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>>> know
> > > > > > > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique
> > > > provient
> > > > > > > >> > >>> dâun
> > > > > > > >> > >>>>>>>>> expéditeur
> > > > > > > >> > >>>>>>>>>>>>>>> externe.
> > > > > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez
> > aucune
> > > > > piĂšce
> > > > > > > >> > >>>>>> jointe
> > > > > > > >> > >>>>>>> si
> > > > > > > >> > >>>>>>>>> vous
> > > > > > > >> > >>>>>>>>>>> ne
> > > > > > > >> > >>>>>>>>>>>>>>>> pouvez
> > > > > > > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur
> > et
> > > si
> > > > > > > >> > >>> vous
> > > > > > > >> > >>>>>> nâĂȘtes
> > > > > > > >> > >>>>>>>> pas
> > > > > > > >> > >>>>>>>>>>>> certain
> > > > > > > >> > >>>>>>>>>>>>>>>> que
> > > > > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping
> âdagsâ
> > > as a
> > > > > > > >> > >>> term â
> > > > > > > >> > >>>>>>>> getting
> > > > > > > >> > >>>>>>>>> to
> > > > > > > >> > >>>>>>>>>>>>> re-use
> > > > > > > >> > >>>>>>>>>>>>>>>>> puns like
> > https://i.imgflip.com/1xhtwh.jpg
> > > <
> > > > > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either
> > way,
> > > > > both
> > > > > > > >> > >>>>> sides
> > > > > > > >> > >>>>>>> have
> > > > > > > >> > >>>>>>>>> good
> > > > > > > >> > >>>>>>>>>>>>>>> reasons
> > > > > > > >> > >>>>>>>>>>>>>>>>> presented.
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>> -a
> > > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel
> Standish
> > > > > > > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA
> > > <mailto:
> > > > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID>
> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the
> name
> > > > comes
> > > > > > > >> > >>> from,
> > > > > > > >> > >>>>>>> "well,
> > > > > > > >> > >>>>>>>>> no
> > > > > > > >> > >>>>>>>>>>> one
> > > > > > > >> > >>>>>>>>>>>>>>>>> actually
> > > > > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something
> up.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek
> > > Potiuk
> > > > <
> > > > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic
> > graph"
> > > > is
> > > > > > > >> > >>> the
> > > > > > > >> > >>>>>>>>>>> tongue-twister,
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek
> > > > Potiuk <
> > > > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent
> > wrote.
> > > I
> > > > > > > >> > >>> would
> > > > > > > >> > >>>>> very
> > > > > > > >> > >>>>>>>> much
> > > > > > > >> > >>>>>>>>> want
> > > > > > > >> > >>>>>>>>>>>> to
> > > > > > > >> > >>>>>>>>>>>>>>> be
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> able
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining
> it
> > > > > > > >> > >>> further.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at
> a
> > > talk
> > > > > > > >> > >>> it's a
> > > > > > > >> > >>>>>>>>>>>> tongue-twister,
> > > > > > > >> > >>>>>>>>>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>>>> I
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall
> how
> > to
> > > > > > > >> > >>> pronounce
> > > > > > > >> > >>>>>> it
> > > > > > > >> > >>>>>>>>>>> properly.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> J.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent
> > > > Bovenzi
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > > > > > > >> > >>>>>>>> brent@astronomer.io.inva
> > > > > > > >> > >>>>>>>>>> lid>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming
> "DAG"
> > > when
> > > > > > > >> > >>>>> starting
> > > > > > > >> > >>>>>> on
> > > > > > > >> > >>>>>>>>> AIP-38
> > > > > > > >> > >>>>>>>>>>> to
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> modernize
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or
> "workflow"
> > > are
> > > > > more
> > > > > > > >> > >>>>>>>>> descriptive of
> > > > > > > >> > >>>>>>>>>>>> what
> > > > > > > >> > >>>>>>>>>>>>>>>> one
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> is
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed
> Acyclic
> > > > Graph
> > > > > > > >> > >>> is an
> > > > > > > >> > >>>>>>>>>>> implementation
> > > > > > > >> > >>>>>>>>>>>>>>>>> detail.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at
> > > this
> > > > > > > >> > >>> point
> > > > > > > >> > >>>>>> "DAG"
> > > > > > > >> > >>>>>>>> has
> > > > > > > >> > >>>>>>>>>>> become
> > > > > > > >> > >>>>>>>>>>>>>>>> "dag"
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> , a
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in
> > > community
> > > > > > > >> > >>>>>> discussion,
> > > > > > > >> > >>>>>>>>> Airflow
> > > > > > > >> > >>>>>>>>>>>>>>>> Summit
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the
> > UI.
> > > > > Let's
> > > > > > > >> > >>>>>> embrace
> > > > > > > >> > >>>>>>>>> "dag".
> > > > > > > >> > >>>>>>>>>>> A
> > > > > > > >> > >>>>>>>>>>>>>>> user
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> just
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the
> > > > technical
> > > > > > > >> > >>>>> concept
> > > > > > > >> > >>>>>>>>> behind
> > > > > > > >> > >>>>>>>>>>> that
> > > > > > > >> > >>>>>>>>>>>>>>>>> word. I
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > > > > > > >> > >>> refactoring so
> > > > > > > >> > >>>>>>> much
> > > > > > > >> > >>>>>>>>> code,
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack
> > > overflow
> > > > > > > >> > >>>>> questions,
> > > > > > > >> > >>>>>>>> etc.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM
> > Daniel
> > > > > > > >> > >>> Standish
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva
> > > > > <mailto:
> > > > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid>
> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like
> > introducing
> > > a
> > > > > lot
> > > > > > > >> > >>> of
> > > > > > > >> > >>>>>> pain
> > > > > > > >> > >>>>>>>> for
> > > > > > > >> > >>>>>>>>>>>>>>>>> questionable
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic
> to
> > > the
> > > > > > > >> > >>> idea. I
> > > > > > > >> > >>>>>>> agree
> > > > > > > >> > >>>>>>>>> the
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> association
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is
> not
> > > > > helpful.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here
> > > some
> > > > > less
> > > > > > > >> > >>>>>>> invasive
> > > > > > > >> > >>>>>>>>>>>>>>> mitigations.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what
> is
> > > to
> > > > > > > >> > >>>>>> de-emphasize
> > > > > > > >> > >>>>>>>> the
> > > > > > > >> > >>>>>>>>>>> math
> > > > > > > >> > >>>>>>>>>>>>>>> nerd
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in
> docs
> > /
> > > > > > > >> > >>> website /
> > > > > > > >> > >>>>>> etc,
> > > > > > > >> > >>>>>>>>> *never
> > > > > > > >> > >>>>>>>>>>>>>>>> define*
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a
> directed
> > > > > acyclic
> > > > > > > >> > >>>>>> graph.
> > > > > > > >> > >>>>>>>>> Always
> > > > > > > >> > >>>>>>>>>>>>>>> define
> > > > > > > >> > >>>>>>>>>>>>>>>>> it
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks,
> > workflow
> > > > > etc.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part
> of
> > it
> > > > is
> > > > > > > >> > >>> like a
> > > > > > > >> > >>>>>>>>> historical
> > > > > > > >> > >>>>>>>>>>>>>>>>> footnote,
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it
> > > somewhere
> > > > > > > >> > >>> hidden.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase
> > in
> > > > the
> > > > > > > >> > >>> docs
> > > > > > > >> > >>>>> in
> > > > > > > >> > >>>>>>>>> general
> > > > > > > >> > >>>>>>>>>>> e.g.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> writing
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing
> > > "DAG" /
> > > > > > > >> > >>> "DAGs"
> > > > > > > >> > >>>>>> etc.
> > > > > > > >> > >>>>>>>> The
> > > > > > > >> > >>>>>>>>>>>> upper
> > > > > > > >> > >>>>>>>>>>>>>>>> case
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> part
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an
> acronym;
> > > but
> > > > > > > >> > >>> "dag" in
> > > > > > > >> > >>>>>>>>> airlfow is
> > > > > > > >> > >>>>>>>>>>>> just
> > > > > > > >> > >>>>>>>>>>>>>>>> an
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association
> > > with
> > > > > > > >> > >>> "DAGs"
> > > > > > > >> > >>>>> is
> > > > > > > >> > >>>>>>> not
> > > > > > > >> > >>>>>>>>> really
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag"
> in
> > > > > airflow
> > > > > > > >> > >>> is
> > > > > > > >> > >>>>> its
> > > > > > > >> > >>>>>>> own
> > > > > > > >> > >>>>>>>>>>> thing,
> > > > > > > >> > >>>>>>>>>>>> is
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph
> > > (which
> > > > > > > >> > >>> nobody
> > > > > > > >> > >>>>>> knows
> > > > > > > >> > >>>>>>>>> about
> > > > > > > >> > >>>>>>>>>>>>>>>> anyway),
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple
> terms
> > > that
> > > > > > > >> > >>> normal
> > > > > > > >> > >>>>>>> people
> > > > > > > >> > >>>>>>>>>>>>>>> understand.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM
> Jarek
> > > > > Potiuk
> > > > > > > >> > >>> <
> > > > > > > >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > > > > > > >> > >>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do
> > > that
> > > > it
> > > > > > > >> > >>> will
> > > > > > > >> > >>>>> be
> > > > > > > >> > >>>>>>>>> extremely
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I
> > think
> > > > it
> > > > > > > >> > >>> will
> > > > > > > >> > >>>>>> make
> > > > > > > >> > >>>>>>> a
> > > > > > > >> > >>>>>>>>> lot of
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> "google"
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow"
> > searches
> > > > not
> > > > > > > >> > >>>>> finding
> > > > > > > >> > >>>>>>> the
> > > > > > > >> > >>>>>>>>> right
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> answers.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> This
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of
> Airflow -
> > > > > > > >> > >>> besides the
> > > > > > > >> > >>>>>>>>> community
> > > > > > > >> > >>>>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>>> ideas
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> that
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast
> number
> > > of
> > > > > > > >> > >>>>> examples,
> > > > > > > >> > >>>>>>>>> problems
> > > > > > > >> > >>>>>>>>>>> and
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we
> have
> > to
> > > > > > > >> > >>> remember
> > > > > > > >> > >>>>>> that
> > > > > > > >> > >>>>>>>>> all the
> > > > > > > >> > >>>>>>>>>>>> AI
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> trained
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> on
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather
> poorly
> > > > > > > >> > >>> matching
> > > > > > > >> > >>>>>>> queries
> > > > > > > >> > >>>>>>>> of
> > > > > > > >> > >>>>>>>>>>>> people.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I
> > could
> > > > > easily
> > > > > > > >> > >>>>>> switch.
> > > > > > > >> > >>>>>>>>> And if
> > > > > > > >> > >>>>>>>>>>> we
> > > > > > > >> > >>>>>>>>>>>>>>> do
> > > > > > > >> > >>>>>>>>>>>>>>>> -
> > > > > > > >> > >>>>>>>>>>>>>>>>>>> I
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or
> > > pipeline
> > > > > > > >> > >>> instead
> > > > > > > >> > >>>>> of
> > > > > > > >> > >>>>>>>>> `dag` if
> > > > > > > >> > >>>>>>>>>>>> not
> > > > > > > >> > >>>>>>>>>>>>>>>> the
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> above
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with
> > > Igor
> > > > > > > >> > >>> that it
> > > > > > > >> > >>>>>>> might
> > > > > > > >> > >>>>>>>>> cause
> > > > > > > >> > >>>>>>>>>>>> more
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> problems
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if
> > others
> > > > will
> > > > > > > >> > >>> think
> > > > > > > >> > >>>>>>> it's
> > > > > > > >> > >>>>>>>> a
> > > > > > > >> > >>>>>>>>> good
> > > > > > > >> > >>>>>>>>>>>>>>>> idea, I
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>> am
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM
> > > Abhishek
> > > > > > > >> > >>> Bhakat
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> <abhishek.bhakat@astronomer.io.inva
> > > > > > > >> > >>> <mailto:
> > > > > > > >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid>
> wrote:
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes
> very
> > > > less
> > > > > > > >> > >>> sense
> > > > > > > >> > >>>>> to
> > > > > > > >> > >>>>>>>>> someone
> > > > > > > >> > >>>>>>>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Brian Greene <br...@heisenbergwoodworking.com>.
I disagree with renaming it, or removing the logical idea of directed
acyclic graph, or redefining the acronym. Itâs less âmathâ to me, and more
âdata structureâ, and a lot of things work the way they do because a dag IS
a DAG.
Whatâs the problem with calling something EXACTLY what it is? So rarely
does a technical team get it so right⊠this feels like a discussion with
marketing people, wanting to obfuscate something completely reasonable, for
no functional value.
How much energy can be spent making this scenario more common? âwhatâs DAG
vs Dag vs Oh you mean DaaG :-(⊠or whateverâ. And why would one want to?
Brian
On Tue, Feb 18, 2025 at 11:45âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> > -0.5
> > Even though the new UI prefers "Dag", I don't think that means "DAG" is
> incorrect.
>
> I think we have a bunch of related things discussed here - I am not exactly
> sure what the -0.5 in this context means Brent. Would it be possible to
> clarify ?
>
> On Tue, Feb 18, 2025 at 6:01âŻPM Ambika Garg <am...@gmail.com>
> wrote:
>
> > +1 would vote for
> > Airflow's *D*ynamic *A*ction *G*raph (DAG)
> >
> > *Dynamic*: Reflects the capability of Airflow to handle workflows that
> > arenât strictly acyclic, but can adapt to changes, iterations and loops
> > within broader context of workflow.
> >
> > *Action*: Emphasizes that airflow is fundamentally about orchestrating
> > actions or tasks. Whether its data processing, model training or any
> other
> > form of computation. It also conveys operational aspect of what Airflow
> > does.
> >
> > *Graph*: Maintains the core concept that workflows in Airflow are
> > visualized and managed as graphs. This term keeps the idea of
> dependencies
> > and relationships between tasks intact.
> >
> > Ambika.
> >
> > On Tue, Feb 18, 2025 at 11:46âŻAM Brent Bovenzi
> <brent@astronomer.io.invalid
> > >
> > wrote:
> >
> > > -0.5
> > >
> > > Even though the new UI prefers "Dag", I don't think that means "DAG" is
> > > incorrect.
> > >
> > > On Tue, Feb 18, 2025 at 11:36âŻAM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > > > I love "Dynamic Airflow Graph" +10 to that Constance.
> > > >
> > > > And yes. I fully agree with your "somewhat" acyclic statement. I saw
> > > people
> > > > mentioning "event-driven" workflows to better describe the AI/ML, but
> > > that
> > > > for me is a synonym of "completely unmanageable workflows" (similar
> to
> > > > micro-services) - what I was alluding to was exactly this - something
> > > like
> > > > curren DAG with parts of it being potentially cyclic - so we are
> > > completely
> > > > aligned here :). And that's what make it "not DAG - in mathematical
> > > sense"
> > > > - even more reason to depart from it,
> > > >
> > > > J.
> > > >
> > > > On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
> > > > <co...@astronomer.io.invalid> wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > I appreciate the discussion about how we position the concept of a
> > DAG
> > > in
> > > > > Airflow. While I agree with updating documentation to de-emphasize
> > the
> > > > > strict mathematical definition, I do not think we should change the
> > > > public
> > > > > interface to use "dag" instead of "DAG" everywhere. It adds
> > > > > unnecessary churn and breaks consistency across the ecosystem.
> > > > >
> > > > > Regarding the broader discussion on whether DAGs in Airflow should
> > > remain
> > > > > acyclic: I acknowledge that some workflows, particularly in ML and
> > AI,
> > > > > require cyclical behaviour (example: iterative refinement until a
> > > > condition
> > > > > is met). However, I think it's important to recognize that even in
> > > those
> > > > > cases, the workflow itself still has an overall start and end. It's
> > not
> > > > an
> > > > > unbounded loop but rather a controlled execution process that
> happens
> > > to
> > > > > have cycles *within* it.
> > > > >
> > > > > That being said, I do see value in reinterpreting "DAG" to better
> > align
> > > > > with how Airflow is actually used, and love shifting away from
> > > "Directed
> > > > > Acyclic Graph" in favour of something more intuitive. However "Data
> > > > Airflow
> > > > > Graph" makes it sound like we're defining Airflow itself rather
> than
> > > > > describing the structure of workflows within it.
> > > > >
> > > > > If we want a new meaning for DAG that resonates with our base while
> > > > keeping
> > > > > Airflow's identity clear, I am proposing three alternatives:
> > > > >
> > > > >
> > > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while
> > > emphasizing
> > > > > flexibility
> > > > > 2. *Distributed Automation Graph*: Highlights execution across
> > > systems
> > > > > and automations without tying it strictly to data
> > > > > 3. *Directed Activity Graph*: Retains the useful meaning of
> > > "Directed"
> > > > > while shifting away from strict acyclic constraints. (My
> personal
> > > > > favourite)
> > > > >
> > > > > I think any shift in terminology should be driven by what best
> > > > communicates
> > > > > Airflow's role workflow automation, rather than just distancing
> > > ourselves
> > > > > from the mathematical definition.
> > > > >
> > > > > Curious to hear more thoughts!
> > > > >
> > > > > On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >
> > > > > > Sorry for the few typos ... My slight dyslexia did not help and
> > > > > autocorrect
> > > > > > did not help either this time : dept -> debt, routed -> rooted
> and
> > a
> > > > few
> > > > > > other small typos :) .
> > > > > >
> > > > > > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > > >
> > > > > > > First of all - yes, agree with Lee and Ash, I think we do not
> > need
> > > to
> > > > > > > change the public interface, I'd also be -0 on that, but docs,
> > > > internal
> > > > > > > strings, yes we could.
> > > > > > >
> > > > > > > But .... After thinking a bit and looking at the discussion
> here,
> > > > where
> > > > > > > people have concerns, I have an alternative take. And I know
> this
> > > > might
> > > > > > be
> > > > > > > controversial, and some people might find it confusing, but why
> > > don't
> > > > > we
> > > > > > > come up with a new acronym for DAG, one that will be more
> airflow
> > > and
> > > > > > less
> > > > > > > "mathematics/algorithm". And I have, I think a good reason for
> > it,
> > > > and
> > > > > I
> > > > > > > even have a proposal. Bear with me,
> > > > > > >
> > > > > > > DAG => *Data Airflow Graph*
> > > > > > >
> > > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1
> now
> > > > (with
> > > > > > > exception of some niche products - you know which one I speak
> > > about -
> > > > > > which
> > > > > > > are unfortunate enough to use the "Dag" in their name - I will
> > > > explain
> > > > > > why
> > > > > > > this is unfortunate, IMHO).
> > > > > > >
> > > > > > > We are lucky to have Airflow for "A" in the acronym and we can
> > make
> > > > it
> > > > > > > even more "synonymous" -> Airflow <> DAG with (A) being
> Airflow.
> > > > > > >
> > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > > > > > >
> > > > > > > Not only because "Directed" and "Acyclic" are mostly not in the
> > > > > > vocabulary
> > > > > > > of most of our users - as Ryan, starting this thread explained
> > > yes, I
> > > > > > agree
> > > > > > > with it, but also I think there is a more important reason - I
> > > > simply
> > > > > > > believe in the future our DAGs might not (and likely will not)
> be
> > > > > > "acyclic".
> > > > > > >
> > > > > > > I know for some it might be herezy and we have not discussed
> any
> > of
> > > > > that
> > > > > > > yet, but more and more I keep on hearing that Machine Learning
> > > > > Workflows
> > > > > > of
> > > > > > > the future are often somewhat cyclic. In our case there is one
> > type
> > > > of
> > > > > > > repeat-in-cycle behaviour - we have "task retry on failure" -
> but
> > > > this
> > > > > is
> > > > > > > only on failure and it basically mens "idempotent" retry - we
> > want
> > > to
> > > > > > redo
> > > > > > > something and get single result out of that and basically
> discard
> > > the
> > > > > > > previous result. But machine learning/AI workflows are
> > different. I
> > > > > had a
> > > > > > > lot of conversations recently (I even co-organized "AI
> low-level
> > > > > > > engineering and hacking" devroom to learn more about those) -
> > and I
> > > > > keep
> > > > > > on
> > > > > > > hearing that "Repeat a sequence of things several times and
> > refine"
> > > > is
> > > > > > > precisely what AI / ML workflows will need - especially with
> > > Agentic
> > > > > > > workflows - often we have cooperating agents which **could** be
> > > part
> > > > of
> > > > > > the
> > > > > > > same DAG, but currently in Airflow where our DAGs are
> "Acyclic"
> > we
> > > > can
> > > > > > > only express "repeat several time and refine" as a single task
> > that
> > > > > will
> > > > > > do
> > > > > > > such refining. If we would like to split such an "agent" doing
> > > their
> > > > > own
> > > > > > > work in "cycles" and communicating with other "agents" doing
> the
> > > > same -
> > > > > > > represented as a multi-task-entity, we have no way to express
> it
> > > > using
> > > > > > our
> > > > > > > current "acyclic" semantics.
> > > > > > >
> > > > > > > But - other than our implementation makes that assumption
> > > currently -
> > > > > > > there is nothing fundamental with Airflow being able to run
> such
> > > > > > "partially
> > > > > > > cyclic workflows". We have all the pieces, we would have to
> > change
> > > > > > > dependency calculation to allow for cycles, update UI to handle
> > > > cycles
> > > > > > > representation - and we could in theory quite seamlessly,
> support
> > > > > > "cyclic"
> > > > > > > workflows.
> > > > > > >
> > > > > > > I know, I know, that's the first time we hear it and that's
> huge
> > > > > change -
> > > > > > > of course, I do not want to start discussing any details of it,
> > not
> > > > > > before
> > > > > > > Airflow 3.0 is out - but I think we should be open that at some
> > > point
> > > > > of
> > > > > > > time the "Acyclic" property is going to go away, And my
> personal
> > > > > > intuition
> > > > > > > tells me that a year from now we will have acyclic workflows
> > > > (assuming
> > > > > > that
> > > > > > > all the foundational work we do on Airflow 3 will really help
> in
> > > > > > decreasing
> > > > > > > all the technical dept, and will open our minds to new ideas).
> > > > > > >
> > > > > > > That's why I think - the sooner we get rid of "Directed
> Acyclic"
> > -
> > > > the
> > > > > > > better. that will - even for us, maintainer free us from being
> > > routed
> > > > > in
> > > > > > > the acyclic nature of Airflow.
> > > > > > >
> > > > > > > How about we change to "Data Airflow Graph" (or any other good
> > > > acronym
> > > > > > > which has (A)irflow instead of (A)cyclic). The name is just a
> > > > proposal.
> > > > > > >
> > > > > > > Now.. I go hide somewhere as I believe that might cause a
> > > "flamewar"
> > > > > :),
> > > > > > > unless on request of Ryan you already sheathed your swords in
> the
> > > > > > > discussion (which seems everyone did).
> > > > > > >
> > > > > > > But honestly - I would really love what you think of "let's
> just
> > > > break
> > > > > > > with Acyclic NOW" and shape our future by using a different
> > > acronym.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > > > > > ephraimanierobi@gmail.com> wrote:
> > > > > > >
> > > > > > >> When I started contributing to Airflow, I had to read up on
> DAG,
> > > and
> > > > > > now I
> > > > > > >> know what it means. I wonder if we are about to have users who
> > use
> > > > the
> > > > > > >> term
> > > > > > >> Dag without knowing that there's a DAG.
> > > > > > >>
> > > > > > >> Slightly concerned.
> > > > > > >>
> > > > > > >> -ephraim
> > > > > > >>
> > > > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com>
> > > wrote:
> > > > > > >>
> > > > > > >> > For user-facing things (e.g., `from airflow import DAG`),
> Iâm
> > > more
> > > > > > like
> > > > > > >> > -0. But for documentation, docstring, internal things and
> > etc.,
> > > we
> > > > > > >> probably
> > > > > > >> > could still change most of them?
> > > > > > >> >
> > > > > > >> > Best,
> > > > > > >> > Wei
> > > > > > >> >
> > > > > > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <
> > > ash@apache.org>
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > I also donât personally think itâs worth the pain (not to
> > > > mention
> > > > > > >> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5
> on
> > > > that.
> > > > > > >> > >
> > > > > > >> > > -ash
> > > > > > >> > >
> > > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > > > > > daniel.imberman@gmail.com
> > > > > > >> >
> > > > > > >> > wrote:
> > > > > > >> > >>
> > > > > > >> > >> I think my biggest concern is a marketing one and not a
> > > > technical
> > > > > > >> one.
> > > > > > >> > >>
> > > > > > >> > >> As has been mentioned on the thread the terms airflow and
> > dag
> > > > are
> > > > > > >> kind
> > > > > > >> > of
> > > > > > >> > >> synonymous and I certainly donât want to give the
> > impression
> > > > that
> > > > > > we
> > > > > > >> are
> > > > > > >> > >> breaking more than we are breaking.
> > > > > > >> > >>
> > > > > > >> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> > > > > > >> > >>
> > > > > > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <
> weilee@apache.org
> > >
> > > > > wrote:
> > > > > > >> > >>
> > > > > > >> > >>> Iâm not sure about adding ruff rules here đ€ I think
> ruff
> > > > rules
> > > > > > are
> > > > > > >> > best
> > > > > > >> > >>> suited for user-facing things but not the airflow code
> > base
> > > > > > itself.
> > > > > > >> If
> > > > > > >> > what
> > > > > > >> > >>> we mean is adding a rule to avoid users using "DAG"
> > *after*
> > > we
> > > > > > >> rename
> > > > > > >> > it,
> > > > > > >> > >>> it's definitely a +1000.
> > > > > > >> > >>>
> > > > > > >> > >>> I just created GitHub issues for this removing "DAG"
> > > > > > >> > >>>
> > > > > > >> > >>> https://github.com/apache/airflow/issues/46842
> > > > > > >> > >>> https://github.com/apache/airflow/issues/46843
> > > > > > >> > >>>
> > > > > > >> > >>> One thing I'm not sure about is whether we want to get
> rid
> > > of
> > > > > > `from
> > > > > > >> > >>> airflow import DAG` as well. đ€
> > > > > > >> > >>>
> > > > > > >> > >>> Best,
> > > > > > >> > >>> Wei
> > > > > > >> > >>>
> > > > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > > > > > >> > >>>> Hard to say until it's looked at :)
> > > > > > >> > >>>>
> > > > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > > > > > >> aritrabasu1999@gmail.com
> > > > > > >> > >
> > > > > > >> > >>>> wrote:
> > > > > > >> > >>>>
> > > > > > >> > >>>>> I can take it up, it's mostly just a doc update right?
> > Or
> > > > are
> > > > > we
> > > > > > >> > doing
> > > > > > >> > >>> code
> > > > > > >> > >>>>> files replacement too?
> > > > > > >> > >>>>> --
> > > > > > >> > >>>>> Regards,
> > > > > > >> > >>>>> Aritra Basu
> > > > > > >> > >>>>>
> > > > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <
> > > > jarek@potiuk.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > >>>>>
> > > > > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ?
> > Any
> > > > > > >> volunteers
> > > > > > >> > >>> to
> > > > > > >> > >>>>>> make it ? It's fun.
> > > > > > >> > >>>>>>
> > > > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > > > > > droiddev510e@gmail.com>
> > > > > > >> > >>> wrote:
> > > > > > >> > >>>>>>
> > > > > > >> > >>>>>>> +1 for ruff rules :)
> > > > > > >> > >>>>>>>
> > > > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace
> 'DAG'
> > > in
> > > > > the
> > > > > > >> > >>> Airflow
> > > > > > >> > >>>>>>> docs, in line with the new UI changes and to make
> the
> > > > > renaming
> > > > > > >> > >>>>>>> consistent across user-facing pages.
> > > > > > >> > >>>>>>>
> > > > > > >> > >>>>>>> Regards,
> > > > > > >> > >>>>>>> Omkar
> > > > > > >> > >>>>>>>
> > > > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > > > > > jarek@potiuk.com>
> > > > > > >> > >>> wrote:
> > > > > > >> > >>>>>>>
> > > > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> > > > > > >> > >>>>>>>>
> > > > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <
> > > > weilee@apache.org
> > > > > >
> > > > > > >> > >>> wrote:
> > > > > > >> > >>>>>>>>
> > > > > > >> > >>>>>>>>> It seems that our current conclusion is to use
> "dag"
> > > or
> > > > > > "Dag"
> > > > > > >> > >>>>> instead
> > > > > > >> > >>>>>>> of
> > > > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all
> "DAG"
> > > in
> > > > > the
> > > > > > >> > >>>>> codebase
> > > > > > >> > >>>>>>> with
> > > > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which
> it
> > > > might
> > > > > be
> > > > > > >> > >>> đ€),
> > > > > > >> > >>>>>>> should
> > > > > > >> > >>>>>>>> we
> > > > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the
> > following
> > > > PRs?
> > > > > > >> > >>>>>>>>>
> > > > > > >> > >>>>>>>>> Best,
> > > > > > >> > >>>>>>>>> Wei
> > > > > > >> > >>>>>>>>>
> > > > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0
> > UI:
> > > > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > > > > > >> > >>>>>>>>>> Let me know if that's the direction we want to
> go.
> > > > > > >> > >>>>>>>>>>
> > > > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo,
> > Rajeshwar
> > > > > > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > > > > > >> > >>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag"
> > is
> > > > > > >> > >>> synonymous
> > > > > > >> > >>>>>> with
> > > > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and
> > > > explain
> > > > > > >> > >>> what
> > > > > > >> > >>>>>>> Airflow
> > > > > > >> > >>>>>>>>> is
> > > > > > >> > >>>>>>>>>>> all about. Although folks would ask, I donât
> ever
> > > use
> > > > > the
> > > > > > >> > >>>>>>>> mathematical
> > > > > > >> > >>>>>>>>>>> definition of DAG.
> > > > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing
> for
> > > many
> > > > > > >> > >>> users
> > > > > > >> > >>>>>> making
> > > > > > >> > >>>>>>>>> such a
> > > > > > >> > >>>>>>>>>>> change - I would rather direct that energy to
> > > > appending
> > > > > > the
> > > > > > >> > >>>>>> Oxford
> > > > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to
> > > > workflows
> > > > > in
> > > > > > >> > >>>>>> Airflow.
> > > > > > >> > >>>>>>>> __
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> -- Rajesh
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> > > > > jarek@potiuk.com
> > > > > > >> > >>>>>> <mailto:
> > > > > > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of
> the
> > > > > > >> > >>>>> organization.
> > > > > > >> > >>>>>> Do
> > > > > > >> > >>>>>>>> not
> > > > > > >> > >>>>>>>>>>> click links or open attachments unless you can
> > > confirm
> > > > > the
> > > > > > >> > >>>>> sender
> > > > > > >> > >>>>>>> and
> > > > > > >> > >>>>>>>>> know
> > > > > > >> > >>>>>>>>>>> the content is safe.
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> > > dâun
> > > > > > >> > >>>>> expéditeur
> > > > > > >> > >>>>>>>>> externe.
> > > > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune
> piĂšce
> > > > > jointe
> > > > > > >> > >>> si
> > > > > > >> > >>>>> vous
> > > > > > >> > >>>>>>> ne
> > > > > > >> > >>>>>>>>> pouvez
> > > > > > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> > vous
> > > > > nâĂȘtes
> > > > > > >> > >>> pas
> > > > > > >> > >>>>>>>> certain
> > > > > > >> > >>>>>>>>> que
> > > > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> From Guido's post:
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have
> been
> > > > > > renamed".
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> Naming is hard.
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> J.
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > > > > >> > >>>>> droiddev510e@gmail.com
> > > > > > >> > >>>>>>>>> <mailto:
> > > > > > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag
> is
> > > an
> > > > > > >> > >>> Airflow
> > > > > > >> > >>>>>> term
> > > > > > >> > >>>>>>>> now
> > > > > > >> > >>>>>>>>>>>> rather
> > > > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe
> it
> > or
> > > > > not,
> > > > > > >> > >>> I've
> > > > > > >> > >>>>>>>> worked
> > > > > > >> > >>>>>>>>> with
> > > > > > >> > >>>>>>>>>>>> work
> > > > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for
> years
> > > now
> > > > > > >> > >>> and are
> > > > > > >> > >>>>>> pro
> > > > > > >> > >>>>>>>>> devs
> > > > > > >> > >>>>>>>>>>> but
> > > > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a
> DAG!
> > > For
> > > > > > >> > >>> them,
> > > > > > >> > >>>>> dag
> > > > > > >> > >>>>>> =
> > > > > > >> > >>>>>>>>> Airflow.
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better
> terms
> > > but
> > > > > > >> > >>> will be
> > > > > > >> > >>>>> a
> > > > > > >> > >>>>>>>> major
> > > > > > >> > >>>>>>>>>>>> behavior
> > > > > > >> > >>>>>>>>>>>> change for the developer community, so we'll
> > need a
> > > > > solid
> > > > > > >> > >>>>> plan
> > > > > > >> > >>>>>> on
> > > > > > >> > >>>>>>>>> how to
> > > > > > >> > >>>>>>>>>>>> introduce it, when we do.
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about
> > the
> > > > > Great
> > > > > > >> > >>>>>> Renaming
> > > > > > >> > >>>>>>>> in
> > > > > > >> > >>>>>>>>> core
> > > > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could
> > > > probably
> > > > > > >> > >>> get
> > > > > > >> > >>>>>> some
> > > > > > >> > >>>>>>>>>>> learnings
> > > > > > >> > >>>>>>>>>>>> from there? Who knows!
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> Guido's blog (2009):
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>
> > > > > > >> > >>>>>>>
> > > > > > >> > >>>>>
> > > > > > >> > >>>
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > >> > >>>>>>>>>>> <
> > > > > > >> > >>>>>>>>>
> > > > > > >> > >>>>>>>
> > > > > > >> > >>>>>
> > > > > > >> > >>>
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > >> > >>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>
> > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > > >> > >>>>>>> <
> > > > > > >> > >>>>>>>>>>>
> > > > > > >> > >>>>>
> > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > > >
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> Cheers,
> > > > > > >> > >>>>>>>>>>>> Omkar
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > > > > > >> > >>> weilee.rx@gmail.com
> > > > > > >> > >>>>>>>>> <mailto:
> > > > > > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > > > > > >> > >>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as
> an
> > > > > airflow
> > > > > > >> > >>>>> term.
> > > > > > >> > >>>>>>> At
> > > > > > >> > >>>>>>>>> least,
> > > > > > >> > >>>>>>>>>>>>> thatâs how I understood it when I first used
> > > > Airflow.
> > > > > I
> > > > > > >> > >>>>> feel
> > > > > > >> > >>>>>>>>> renaming
> > > > > > >> > >>>>>>>>>>> it
> > > > > > >> > >>>>>>>>>>>> at
> > > > > > >> > >>>>>>>>>>>>> this stage would require considerable effort
> > from
> > > > > > >> > >>>>> maintainers
> > > > > > >> > >>>>>>> and
> > > > > > >> > >>>>>>>>>>>> existing
> > > > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > > > > > >> > >>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>> Best,
> > > > > > >> > >>>>>>>>>>>>> Wei
> > > > > > >> > >>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > > > > >> > >>>>>> kaxilnaik@gmail.com
> > > > > > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we
> > > > re-kindle
> > > > > > >> > >>>>> this
> > > > > > >> > >>>>>>>>> discussion
> > > > > > >> > >>>>>>>>>>>> for
> > > > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause
> too
> > > > much
> > > > > > >> > >>>>>>> disruption
> > > > > > >> > >>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance
> > Martineau
> > > > > > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > > > > > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > > > > > >> > >>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with
> > > Airflow
> > > > > > >> > >>>>>> experience
> > > > > > >> > >>>>>>>>> what a
> > > > > > >> > >>>>>>>>>>>> dag
> > > > > > >> > >>>>>>>>>>>>> is,
> > > > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow
> > attributes
> > > -
> > > > > > >> > >>> stuff
> > > > > > >> > >>>>>> like
> > > > > > >> > >>>>>>>>> dags
> > > > > > >> > >>>>>>>>>>>> being
> > > > > > >> > >>>>>>>>>>>>> a
> > > > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The
> > > > structure
> > > > > > >> > >>>>>> doesn't
> > > > > > >> > >>>>>>>>> come up.
> > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is
> to
> > > > > > >> > >>> embrace
> > > > > > >> > >>>>> the
> > > > > > >> > >>>>>>> name
> > > > > > >> > >>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure
> > aspect.
> > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > > > > > >> > >>>>> vikram@astronomer.io.inva
> > > > > > >> > >>>>>>>>> lid>
> > > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I
> remember
> > > > > > >> > >>>>> struggling
> > > > > > >> > >>>>>>> with
> > > > > > >> > >>>>>>>>> this
> > > > > > >> > >>>>>>>>>>>>> when I
> > > > > > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > > > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it
> > > being
> > > > an
> > > > > > >> > >>>>>>>> established
> > > > > > >> > >>>>>>>>>>>> concept
> > > > > > >> > >>>>>>>>>>>>>>> now
> > > > > > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the
> > perspective
> > > > > > >> > >>> best
> > > > > > >> > >>>>>>>> expressed
> > > > > > >> > >>>>>>>>> by
> > > > > > >> > >>>>>>>>>>>>> Daniel
> > > > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word,
> > > rather
> > > > > > >> > >>> than
> > > > > > >> > >>>>>> the
> > > > > > >> > >>>>>>>>> computer
> > > > > > >> > >>>>>>>>>>>>>>> science
> > > > > > >> > >>>>>>>>>>>>>>>> concept.
> > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>> Best regards,
> > > > > > >> > >>>>>>>>>>>>>>>> Vikram
> > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira,
> > Niko
> > > > > > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > > > > > >> > >>>>>> onikolas@amazon.com.inva
> > > > > > >> > >>>>>>>>> lid
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of:
> > You're
> > > > > > >> > >>> right
> > > > > > >> > >>>>>> Ryan,
> > > > > > >> > >>>>>>>> DAG
> > > > > > >> > >>>>>>>>>>> isn't
> > > > > > >> > >>>>>>>>>>>>>>> great
> > > > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it
> > will
> > > > > > >> > >>> cause
> > > > > > >> > >>>>> much
> > > > > > >> > >>>>>>>> more
> > > > > > >> > >>>>>>>>>>>> wreckage
> > > > > > >> > >>>>>>>>>>>>>>>> than
> > > > > > >> > >>>>>>>>>>>>>>>>> it solves.
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away
> > > from
> > > > > > >> > >>>>> defining
> > > > > > >> > >>>>>>>> DAG.
> > > > > > >> > >>>>>>>>> I
> > > > > > >> > >>>>>>>>>>>> think
> > > > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a
> > community
> > > > > > >> > >>> for a
> > > > > > >> > >>>>>> while
> > > > > > >> > >>>>>>>> now
> > > > > > >> > >>>>>>>>>>>> anyway,
> > > > > > >> > >>>>>>>>>>>>>>> so
> > > > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> Cheers,
> > > > > > >> > >>>>>>>>>>>>>>>>> Niko
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> ________________________________
> > > > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org
> > > > <mailto:
> > > > > > >> > >>>>>>>>> ash@apache.org>>
> > > > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > > >> > >>>>>> dev@airflow.apache.org
> > > > > > >> > >>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should
> deprecate
> > > the
> > > > > > >> > >>> term
> > > > > > >> > >>>>>> "DAG"
> > > > > > >> > >>>>>>>>> for end
> > > > > > >> > >>>>>>>>>>>>>>> users
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from
> outside
> > of
> > > > the
> > > > > > >> > >>>>>>>>> organization.
> > > > > > >> > >>>>>>>>>>> Do
> > > > > > >> > >>>>>>>>>>>>> not
> > > > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you
> > can
> > > > > > >> > >>> confirm
> > > > > > >> > >>>>>> the
> > > > > > >> > >>>>>>>>> sender
> > > > > > >> > >>>>>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>>> know
> > > > > > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique
> > > provient
> > > > > > >> > >>> dâun
> > > > > > >> > >>>>>>>>> expéditeur
> > > > > > >> > >>>>>>>>>>>>>>> externe.
> > > > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez
> aucune
> > > > piĂšce
> > > > > > >> > >>>>>> jointe
> > > > > > >> > >>>>>>> si
> > > > > > >> > >>>>>>>>> vous
> > > > > > >> > >>>>>>>>>>> ne
> > > > > > >> > >>>>>>>>>>>>>>>> pouvez
> > > > > > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur
> et
> > si
> > > > > > >> > >>> vous
> > > > > > >> > >>>>>> nâĂȘtes
> > > > > > >> > >>>>>>>> pas
> > > > > > >> > >>>>>>>>>>>> certain
> > > > > > >> > >>>>>>>>>>>>>>>> que
> > > > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ
> > as a
> > > > > > >> > >>> term â
> > > > > > >> > >>>>>>>> getting
> > > > > > >> > >>>>>>>>> to
> > > > > > >> > >>>>>>>>>>>>> re-use
> > > > > > >> > >>>>>>>>>>>>>>>>> puns like
> https://i.imgflip.com/1xhtwh.jpg
> > <
> > > > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either
> way,
> > > > both
> > > > > > >> > >>>>> sides
> > > > > > >> > >>>>>>> have
> > > > > > >> > >>>>>>>>> good
> > > > > > >> > >>>>>>>>>>>>>>> reasons
> > > > > > >> > >>>>>>>>>>>>>>>>> presented.
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>> -a
> > > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA
> > <mailto:
> > > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name
> > > comes
> > > > > > >> > >>> from,
> > > > > > >> > >>>>>>> "well,
> > > > > > >> > >>>>>>>>> no
> > > > > > >> > >>>>>>>>>>> one
> > > > > > >> > >>>>>>>>>>>>>>>>> actually
> > > > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek
> > Potiuk
> > > <
> > > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic
> graph"
> > > is
> > > > > > >> > >>> the
> > > > > > >> > >>>>>>>>>>> tongue-twister,
> > > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek
> > > Potiuk <
> > > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent
> wrote.
> > I
> > > > > > >> > >>> would
> > > > > > >> > >>>>> very
> > > > > > >> > >>>>>>>> much
> > > > > > >> > >>>>>>>>> want
> > > > > > >> > >>>>>>>>>>>> to
> > > > > > >> > >>>>>>>>>>>>>>> be
> > > > > > >> > >>>>>>>>>>>>>>>>>>> able
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> > > > > > >> > >>> further.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a
> > talk
> > > > > > >> > >>> it's a
> > > > > > >> > >>>>>>>>>>>> tongue-twister,
> > > > > > >> > >>>>>>>>>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>>>> I
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how
> to
> > > > > > >> > >>> pronounce
> > > > > > >> > >>>>>> it
> > > > > > >> > >>>>>>>>>>> properly.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> J.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent
> > > Bovenzi
> > > > > > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > > > > > >> > >>>>>>>> brent@astronomer.io.inva
> > > > > > >> > >>>>>>>>>> lid>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG"
> > when
> > > > > > >> > >>>>> starting
> > > > > > >> > >>>>>> on
> > > > > > >> > >>>>>>>>> AIP-38
> > > > > > >> > >>>>>>>>>>> to
> > > > > > >> > >>>>>>>>>>>>>>>>>>> modernize
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow"
> > are
> > > > more
> > > > > > >> > >>>>>>>>> descriptive of
> > > > > > >> > >>>>>>>>>>>> what
> > > > > > >> > >>>>>>>>>>>>>>>> one
> > > > > > >> > >>>>>>>>>>>>>>>>>>> is
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic
> > > Graph
> > > > > > >> > >>> is an
> > > > > > >> > >>>>>>>>>>> implementation
> > > > > > >> > >>>>>>>>>>>>>>>>> detail.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at
> > this
> > > > > > >> > >>> point
> > > > > > >> > >>>>>> "DAG"
> > > > > > >> > >>>>>>>> has
> > > > > > >> > >>>>>>>>>>> become
> > > > > > >> > >>>>>>>>>>>>>>>> "dag"
> > > > > > >> > >>>>>>>>>>>>>>>>>>> , a
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in
> > community
> > > > > > >> > >>>>>> discussion,
> > > > > > >> > >>>>>>>>> Airflow
> > > > > > >> > >>>>>>>>>>>>>>>> Summit
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the
> UI.
> > > > Let's
> > > > > > >> > >>>>>> embrace
> > > > > > >> > >>>>>>>>> "dag".
> > > > > > >> > >>>>>>>>>>> A
> > > > > > >> > >>>>>>>>>>>>>>> user
> > > > > > >> > >>>>>>>>>>>>>>>>>>> just
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the
> > > technical
> > > > > > >> > >>>>> concept
> > > > > > >> > >>>>>>>>> behind
> > > > > > >> > >>>>>>>>>>> that
> > > > > > >> > >>>>>>>>>>>>>>>>> word. I
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > > > > > >> > >>> refactoring so
> > > > > > >> > >>>>>>> much
> > > > > > >> > >>>>>>>>> code,
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack
> > overflow
> > > > > > >> > >>>>> questions,
> > > > > > >> > >>>>>>>> etc.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM
> Daniel
> > > > > > >> > >>> Standish
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva
> > > > <mailto:
> > > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like
> introducing
> > a
> > > > lot
> > > > > > >> > >>> of
> > > > > > >> > >>>>>> pain
> > > > > > >> > >>>>>>>> for
> > > > > > >> > >>>>>>>>>>>>>>>>> questionable
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to
> > the
> > > > > > >> > >>> idea. I
> > > > > > >> > >>>>>>> agree
> > > > > > >> > >>>>>>>>> the
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> association
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not
> > > > helpful.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here
> > some
> > > > less
> > > > > > >> > >>>>>>> invasive
> > > > > > >> > >>>>>>>>>>>>>>> mitigations.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is
> > to
> > > > > > >> > >>>>>> de-emphasize
> > > > > > >> > >>>>>>>> the
> > > > > > >> > >>>>>>>>>>> math
> > > > > > >> > >>>>>>>>>>>>>>> nerd
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs
> /
> > > > > > >> > >>> website /
> > > > > > >> > >>>>>> etc,
> > > > > > >> > >>>>>>>>> *never
> > > > > > >> > >>>>>>>>>>>>>>>> define*
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed
> > > > acyclic
> > > > > > >> > >>>>>> graph.
> > > > > > >> > >>>>>>>>> Always
> > > > > > >> > >>>>>>>>>>>>>>> define
> > > > > > >> > >>>>>>>>>>>>>>>>> it
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks,
> workflow
> > > > etc.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of
> it
> > > is
> > > > > > >> > >>> like a
> > > > > > >> > >>>>>>>>> historical
> > > > > > >> > >>>>>>>>>>>>>>>>> footnote,
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it
> > somewhere
> > > > > > >> > >>> hidden.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase
> in
> > > the
> > > > > > >> > >>> docs
> > > > > > >> > >>>>> in
> > > > > > >> > >>>>>>>>> general
> > > > > > >> > >>>>>>>>>>> e.g.
> > > > > > >> > >>>>>>>>>>>>>>>>>>> writing
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing
> > "DAG" /
> > > > > > >> > >>> "DAGs"
> > > > > > >> > >>>>>> etc.
> > > > > > >> > >>>>>>>> The
> > > > > > >> > >>>>>>>>>>>> upper
> > > > > > >> > >>>>>>>>>>>>>>>> case
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> part
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym;
> > but
> > > > > > >> > >>> "dag" in
> > > > > > >> > >>>>>>>>> airlfow is
> > > > > > >> > >>>>>>>>>>>> just
> > > > > > >> > >>>>>>>>>>>>>>>> an
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association
> > with
> > > > > > >> > >>> "DAGs"
> > > > > > >> > >>>>> is
> > > > > > >> > >>>>>>> not
> > > > > > >> > >>>>>>>>> really
> > > > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in
> > > > airflow
> > > > > > >> > >>> is
> > > > > > >> > >>>>> its
> > > > > > >> > >>>>>>> own
> > > > > > >> > >>>>>>>>>>> thing,
> > > > > > >> > >>>>>>>>>>>> is
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph
> > (which
> > > > > > >> > >>> nobody
> > > > > > >> > >>>>>> knows
> > > > > > >> > >>>>>>>>> about
> > > > > > >> > >>>>>>>>>>>>>>>> anyway),
> > > > > > >> > >>>>>>>>>>>>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms
> > that
> > > > > > >> > >>> normal
> > > > > > >> > >>>>>>> people
> > > > > > >> > >>>>>>>>>>>>>>> understand.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek
> > > > Potiuk
> > > > > > >> > >>> <
> > > > > > >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > > > > > >> > >>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do
> > that
> > > it
> > > > > > >> > >>> will
> > > > > > >> > >>>>> be
> > > > > > >> > >>>>>>>>> extremely
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I
> think
> > > it
> > > > > > >> > >>> will
> > > > > > >> > >>>>>> make
> > > > > > >> > >>>>>>> a
> > > > > > >> > >>>>>>>>> lot of
> > > > > > >> > >>>>>>>>>>>>>>>>>>> "google"
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow"
> searches
> > > not
> > > > > > >> > >>>>> finding
> > > > > > >> > >>>>>>> the
> > > > > > >> > >>>>>>>>> right
> > > > > > >> > >>>>>>>>>>>>>>>>>>> answers.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> This
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> > > > > > >> > >>> besides the
> > > > > > >> > >>>>>>>>> community
> > > > > > >> > >>>>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>>> ideas
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> that
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number
> > of
> > > > > > >> > >>>>> examples,
> > > > > > >> > >>>>>>>>> problems
> > > > > > >> > >>>>>>>>>>> and
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have
> to
> > > > > > >> > >>> remember
> > > > > > >> > >>>>>> that
> > > > > > >> > >>>>>>>>> all the
> > > > > > >> > >>>>>>>>>>>> AI
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> trained
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> on
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> > > > > > >> > >>> matching
> > > > > > >> > >>>>>>> queries
> > > > > > >> > >>>>>>>> of
> > > > > > >> > >>>>>>>>>>>> people.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I
> could
> > > > easily
> > > > > > >> > >>>>>> switch.
> > > > > > >> > >>>>>>>>> And if
> > > > > > >> > >>>>>>>>>>> we
> > > > > > >> > >>>>>>>>>>>>>>> do
> > > > > > >> > >>>>>>>>>>>>>>>> -
> > > > > > >> > >>>>>>>>>>>>>>>>>>> I
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or
> > pipeline
> > > > > > >> > >>> instead
> > > > > > >> > >>>>> of
> > > > > > >> > >>>>>>>>> `dag` if
> > > > > > >> > >>>>>>>>>>>> not
> > > > > > >> > >>>>>>>>>>>>>>>> the
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> above
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with
> > Igor
> > > > > > >> > >>> that it
> > > > > > >> > >>>>>>> might
> > > > > > >> > >>>>>>>>> cause
> > > > > > >> > >>>>>>>>>>>> more
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> problems
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if
> others
> > > will
> > > > > > >> > >>> think
> > > > > > >> > >>>>>>> it's
> > > > > > >> > >>>>>>>> a
> > > > > > >> > >>>>>>>>> good
> > > > > > >> > >>>>>>>>>>>>>>>> idea, I
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>> am
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM
> > Abhishek
> > > > > > >> > >>> Bhakat
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> > > > > > >> > >>> <mailto:
> > > > > > >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very
> > > less
> > > > > > >> > >>> sense
> > > > > > >> > >>>>> to
> > > > > > >> > >>>>>>>>> someone
> > > > > > >> > >>>>>>>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
> -0.5
> Even though the new UI prefers "Dag", I don't think that means "DAG" is
incorrect.
I think we have a bunch of related things discussed here - I am not exactly
sure what the -0.5 in this context means Brent. Would it be possible to
clarify ?
On Tue, Feb 18, 2025 at 6:01âŻPM Ambika Garg <am...@gmail.com>
wrote:
> +1 would vote for
> Airflow's *D*ynamic *A*ction *G*raph (DAG)
>
> *Dynamic*: Reflects the capability of Airflow to handle workflows that
> arenât strictly acyclic, but can adapt to changes, iterations and loops
> within broader context of workflow.
>
> *Action*: Emphasizes that airflow is fundamentally about orchestrating
> actions or tasks. Whether its data processing, model training or any other
> form of computation. It also conveys operational aspect of what Airflow
> does.
>
> *Graph*: Maintains the core concept that workflows in Airflow are
> visualized and managed as graphs. This term keeps the idea of dependencies
> and relationships between tasks intact.
>
> Ambika.
>
> On Tue, Feb 18, 2025 at 11:46âŻAM Brent Bovenzi <brent@astronomer.io.invalid
> >
> wrote:
>
> > -0.5
> >
> > Even though the new UI prefers "Dag", I don't think that means "DAG" is
> > incorrect.
> >
> > On Tue, Feb 18, 2025 at 11:36âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I love "Dynamic Airflow Graph" +10 to that Constance.
> > >
> > > And yes. I fully agree with your "somewhat" acyclic statement. I saw
> > people
> > > mentioning "event-driven" workflows to better describe the AI/ML, but
> > that
> > > for me is a synonym of "completely unmanageable workflows" (similar to
> > > micro-services) - what I was alluding to was exactly this - something
> > like
> > > curren DAG with parts of it being potentially cyclic - so we are
> > completely
> > > aligned here :). And that's what make it "not DAG - in mathematical
> > sense"
> > > - even more reason to depart from it,
> > >
> > > J.
> > >
> > > On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
> > > <co...@astronomer.io.invalid> wrote:
> > >
> > > > Hey all,
> > > >
> > > > I appreciate the discussion about how we position the concept of a
> DAG
> > in
> > > > Airflow. While I agree with updating documentation to de-emphasize
> the
> > > > strict mathematical definition, I do not think we should change the
> > > public
> > > > interface to use "dag" instead of "DAG" everywhere. It adds
> > > > unnecessary churn and breaks consistency across the ecosystem.
> > > >
> > > > Regarding the broader discussion on whether DAGs in Airflow should
> > remain
> > > > acyclic: I acknowledge that some workflows, particularly in ML and
> AI,
> > > > require cyclical behaviour (example: iterative refinement until a
> > > condition
> > > > is met). However, I think it's important to recognize that even in
> > those
> > > > cases, the workflow itself still has an overall start and end. It's
> not
> > > an
> > > > unbounded loop but rather a controlled execution process that happens
> > to
> > > > have cycles *within* it.
> > > >
> > > > That being said, I do see value in reinterpreting "DAG" to better
> align
> > > > with how Airflow is actually used, and love shifting away from
> > "Directed
> > > > Acyclic Graph" in favour of something more intuitive. However "Data
> > > Airflow
> > > > Graph" makes it sound like we're defining Airflow itself rather than
> > > > describing the structure of workflows within it.
> > > >
> > > > If we want a new meaning for DAG that resonates with our base while
> > > keeping
> > > > Airflow's identity clear, I am proposing three alternatives:
> > > >
> > > >
> > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while
> > emphasizing
> > > > flexibility
> > > > 2. *Distributed Automation Graph*: Highlights execution across
> > systems
> > > > and automations without tying it strictly to data
> > > > 3. *Directed Activity Graph*: Retains the useful meaning of
> > "Directed"
> > > > while shifting away from strict acyclic constraints. (My personal
> > > > favourite)
> > > >
> > > > I think any shift in terminology should be driven by what best
> > > communicates
> > > > Airflow's role workflow automation, rather than just distancing
> > ourselves
> > > > from the mathematical definition.
> > > >
> > > > Curious to hear more thoughts!
> > > >
> > > > On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >
> > > > > Sorry for the few typos ... My slight dyslexia did not help and
> > > > autocorrect
> > > > > did not help either this time : dept -> debt, routed -> rooted and
> a
> > > few
> > > > > other small typos :) .
> > > > >
> > > > > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >
> > > > > > First of all - yes, agree with Lee and Ash, I think we do not
> need
> > to
> > > > > > change the public interface, I'd also be -0 on that, but docs,
> > > internal
> > > > > > strings, yes we could.
> > > > > >
> > > > > > But .... After thinking a bit and looking at the discussion here,
> > > where
> > > > > > people have concerns, I have an alternative take. And I know this
> > > might
> > > > > be
> > > > > > controversial, and some people might find it confusing, but why
> > don't
> > > > we
> > > > > > come up with a new acronym for DAG, one that will be more airflow
> > and
> > > > > less
> > > > > > "mathematics/algorithm". And I have, I think a good reason for
> it,
> > > and
> > > > I
> > > > > > even have a proposal. Bear with me,
> > > > > >
> > > > > > DAG => *Data Airflow Graph*
> > > > > >
> > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now
> > > (with
> > > > > > exception of some niche products - you know which one I speak
> > about -
> > > > > which
> > > > > > are unfortunate enough to use the "Dag" in their name - I will
> > > explain
> > > > > why
> > > > > > this is unfortunate, IMHO).
> > > > > >
> > > > > > We are lucky to have Airflow for "A" in the acronym and we can
> make
> > > it
> > > > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow.
> > > > > >
> > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > > > > >
> > > > > > Not only because "Directed" and "Acyclic" are mostly not in the
> > > > > vocabulary
> > > > > > of most of our users - as Ryan, starting this thread explained
> > yes, I
> > > > > agree
> > > > > > with it, but also I think there is a more important reason - I
> > > simply
> > > > > > believe in the future our DAGs might not (and likely will not) be
> > > > > "acyclic".
> > > > > >
> > > > > > I know for some it might be herezy and we have not discussed any
> of
> > > > that
> > > > > > yet, but more and more I keep on hearing that Machine Learning
> > > > Workflows
> > > > > of
> > > > > > the future are often somewhat cyclic. In our case there is one
> type
> > > of
> > > > > > repeat-in-cycle behaviour - we have "task retry on failure" - but
> > > this
> > > > is
> > > > > > only on failure and it basically mens "idempotent" retry - we
> want
> > to
> > > > > redo
> > > > > > something and get single result out of that and basically discard
> > the
> > > > > > previous result. But machine learning/AI workflows are
> different. I
> > > > had a
> > > > > > lot of conversations recently (I even co-organized "AI low-level
> > > > > > engineering and hacking" devroom to learn more about those) -
> and I
> > > > keep
> > > > > on
> > > > > > hearing that "Repeat a sequence of things several times and
> refine"
> > > is
> > > > > > precisely what AI / ML workflows will need - especially with
> > Agentic
> > > > > > workflows - often we have cooperating agents which **could** be
> > part
> > > of
> > > > > the
> > > > > > same DAG, but currently in Airflow where our DAGs are "Acyclic"
> we
> > > can
> > > > > > only express "repeat several time and refine" as a single task
> that
> > > > will
> > > > > do
> > > > > > such refining. If we would like to split such an "agent" doing
> > their
> > > > own
> > > > > > work in "cycles" and communicating with other "agents" doing the
> > > same -
> > > > > > represented as a multi-task-entity, we have no way to express it
> > > using
> > > > > our
> > > > > > current "acyclic" semantics.
> > > > > >
> > > > > > But - other than our implementation makes that assumption
> > currently -
> > > > > > there is nothing fundamental with Airflow being able to run such
> > > > > "partially
> > > > > > cyclic workflows". We have all the pieces, we would have to
> change
> > > > > > dependency calculation to allow for cycles, update UI to handle
> > > cycles
> > > > > > representation - and we could in theory quite seamlessly, support
> > > > > "cyclic"
> > > > > > workflows.
> > > > > >
> > > > > > I know, I know, that's the first time we hear it and that's huge
> > > > change -
> > > > > > of course, I do not want to start discussing any details of it,
> not
> > > > > before
> > > > > > Airflow 3.0 is out - but I think we should be open that at some
> > point
> > > > of
> > > > > > time the "Acyclic" property is going to go away, And my personal
> > > > > intuition
> > > > > > tells me that a year from now we will have acyclic workflows
> > > (assuming
> > > > > that
> > > > > > all the foundational work we do on Airflow 3 will really help in
> > > > > decreasing
> > > > > > all the technical dept, and will open our minds to new ideas).
> > > > > >
> > > > > > That's why I think - the sooner we get rid of "Directed Acyclic"
> -
> > > the
> > > > > > better. that will - even for us, maintainer free us from being
> > routed
> > > > in
> > > > > > the acyclic nature of Airflow.
> > > > > >
> > > > > > How about we change to "Data Airflow Graph" (or any other good
> > > acronym
> > > > > > which has (A)irflow instead of (A)cyclic). The name is just a
> > > proposal.
> > > > > >
> > > > > > Now.. I go hide somewhere as I believe that might cause a
> > "flamewar"
> > > > :),
> > > > > > unless on request of Ryan you already sheathed your swords in the
> > > > > > discussion (which seems everyone did).
> > > > > >
> > > > > > But honestly - I would really love what you think of "let's just
> > > break
> > > > > > with Acyclic NOW" and shape our future by using a different
> > acronym.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > > > > ephraimanierobi@gmail.com> wrote:
> > > > > >
> > > > > >> When I started contributing to Airflow, I had to read up on DAG,
> > and
> > > > > now I
> > > > > >> know what it means. I wonder if we are about to have users who
> use
> > > the
> > > > > >> term
> > > > > >> Dag without knowing that there's a DAG.
> > > > > >>
> > > > > >> Slightly concerned.
> > > > > >>
> > > > > >> -ephraim
> > > > > >>
> > > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com>
> > wrote:
> > > > > >>
> > > > > >> > For user-facing things (e.g., `from airflow import DAG`), Iâm
> > more
> > > > > like
> > > > > >> > -0. But for documentation, docstring, internal things and
> etc.,
> > we
> > > > > >> probably
> > > > > >> > could still change most of them?
> > > > > >> >
> > > > > >> > Best,
> > > > > >> > Wei
> > > > > >> >
> > > > > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <
> > ash@apache.org>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > I also donât personally think itâs worth the pain (not to
> > > mention
> > > > > >> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on
> > > that.
> > > > > >> > >
> > > > > >> > > -ash
> > > > > >> > >
> > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > > > > daniel.imberman@gmail.com
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> > >>
> > > > > >> > >> I think my biggest concern is a marketing one and not a
> > > technical
> > > > > >> one.
> > > > > >> > >>
> > > > > >> > >> As has been mentioned on the thread the terms airflow and
> dag
> > > are
> > > > > >> kind
> > > > > >> > of
> > > > > >> > >> synonymous and I certainly donât want to give the
> impression
> > > that
> > > > > we
> > > > > >> are
> > > > > >> > >> breaking more than we are breaking.
> > > > > >> > >>
> > > > > >> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> > > > > >> > >>
> > > > > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <weilee@apache.org
> >
> > > > wrote:
> > > > > >> > >>
> > > > > >> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff
> > > rules
> > > > > are
> > > > > >> > best
> > > > > >> > >>> suited for user-facing things but not the airflow code
> base
> > > > > itself.
> > > > > >> If
> > > > > >> > what
> > > > > >> > >>> we mean is adding a rule to avoid users using "DAG"
> *after*
> > we
> > > > > >> rename
> > > > > >> > it,
> > > > > >> > >>> it's definitely a +1000.
> > > > > >> > >>>
> > > > > >> > >>> I just created GitHub issues for this removing "DAG"
> > > > > >> > >>>
> > > > > >> > >>> https://github.com/apache/airflow/issues/46842
> > > > > >> > >>> https://github.com/apache/airflow/issues/46843
> > > > > >> > >>>
> > > > > >> > >>> One thing I'm not sure about is whether we want to get rid
> > of
> > > > > `from
> > > > > >> > >>> airflow import DAG` as well. đ€
> > > > > >> > >>>
> > > > > >> > >>> Best,
> > > > > >> > >>> Wei
> > > > > >> > >>>
> > > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > > > > >> > >>>> Hard to say until it's looked at :)
> > > > > >> > >>>>
> > > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > > > > >> aritrabasu1999@gmail.com
> > > > > >> > >
> > > > > >> > >>>> wrote:
> > > > > >> > >>>>
> > > > > >> > >>>>> I can take it up, it's mostly just a doc update right?
> Or
> > > are
> > > > we
> > > > > >> > doing
> > > > > >> > >>> code
> > > > > >> > >>>>> files replacement too?
> > > > > >> > >>>>> --
> > > > > >> > >>>>> Regards,
> > > > > >> > >>>>> Aritra Basu
> > > > > >> > >>>>>
> > > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <
> > > jarek@potiuk.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > >>>>>
> > > > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ?
> Any
> > > > > >> volunteers
> > > > > >> > >>> to
> > > > > >> > >>>>>> make it ? It's fun.
> > > > > >> > >>>>>>
> > > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > > > > droiddev510e@gmail.com>
> > > > > >> > >>> wrote:
> > > > > >> > >>>>>>
> > > > > >> > >>>>>>> +1 for ruff rules :)
> > > > > >> > >>>>>>>
> > > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG'
> > in
> > > > the
> > > > > >> > >>> Airflow
> > > > > >> > >>>>>>> docs, in line with the new UI changes and to make the
> > > > renaming
> > > > > >> > >>>>>>> consistent across user-facing pages.
> > > > > >> > >>>>>>>
> > > > > >> > >>>>>>> Regards,
> > > > > >> > >>>>>>> Omkar
> > > > > >> > >>>>>>>
> > > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > > > > jarek@potiuk.com>
> > > > > >> > >>> wrote:
> > > > > >> > >>>>>>>
> > > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> > > > > >> > >>>>>>>>
> > > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <
> > > weilee@apache.org
> > > > >
> > > > > >> > >>> wrote:
> > > > > >> > >>>>>>>>
> > > > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag"
> > or
> > > > > "Dag"
> > > > > >> > >>>>> instead
> > > > > >> > >>>>>>> of
> > > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG"
> > in
> > > > the
> > > > > >> > >>>>> codebase
> > > > > >> > >>>>>>> with
> > > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it
> > > might
> > > > be
> > > > > >> > >>> đ€),
> > > > > >> > >>>>>>> should
> > > > > >> > >>>>>>>> we
> > > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the
> following
> > > PRs?
> > > > > >> > >>>>>>>>>
> > > > > >> > >>>>>>>>> Best,
> > > > > >> > >>>>>>>>> Wei
> > > > > >> > >>>>>>>>>
> > > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0
> UI:
> > > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > > > > >> > >>>>>>>>>> Let me know if that's the direction we want to go.
> > > > > >> > >>>>>>>>>>
> > > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo,
> Rajeshwar
> > > > > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > > > > >> > >>>>>>>>>>
> > > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag"
> is
> > > > > >> > >>> synonymous
> > > > > >> > >>>>>> with
> > > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and
> > > explain
> > > > > >> > >>> what
> > > > > >> > >>>>>>> Airflow
> > > > > >> > >>>>>>>>> is
> > > > > >> > >>>>>>>>>>> all about. Although folks would ask, I donât ever
> > use
> > > > the
> > > > > >> > >>>>>>>> mathematical
> > > > > >> > >>>>>>>>>>> definition of DAG.
> > > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for
> > many
> > > > > >> > >>> users
> > > > > >> > >>>>>> making
> > > > > >> > >>>>>>>>> such a
> > > > > >> > >>>>>>>>>>> change - I would rather direct that energy to
> > > appending
> > > > > the
> > > > > >> > >>>>>> Oxford
> > > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to
> > > workflows
> > > > in
> > > > > >> > >>>>>> Airflow.
> > > > > >> > >>>>>>>> __
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> -- Rajesh
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> > > > jarek@potiuk.com
> > > > > >> > >>>>>> <mailto:
> > > > > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the
> > > > > >> > >>>>> organization.
> > > > > >> > >>>>>> Do
> > > > > >> > >>>>>>>> not
> > > > > >> > >>>>>>>>>>> click links or open attachments unless you can
> > confirm
> > > > the
> > > > > >> > >>>>> sender
> > > > > >> > >>>>>>> and
> > > > > >> > >>>>>>>>> know
> > > > > >> > >>>>>>>>>>> the content is safe.
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> > dâun
> > > > > >> > >>>>> expéditeur
> > > > > >> > >>>>>>>>> externe.
> > > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > > > jointe
> > > > > >> > >>> si
> > > > > >> > >>>>> vous
> > > > > >> > >>>>>>> ne
> > > > > >> > >>>>>>>>> pouvez
> > > > > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> vous
> > > > nâĂȘtes
> > > > > >> > >>> pas
> > > > > >> > >>>>>>>> certain
> > > > > >> > >>>>>>>>> que
> > > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> From Guido's post:
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been
> > > > > renamed".
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> Naming is hard.
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> J.
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > > > >> > >>>>> droiddev510e@gmail.com
> > > > > >> > >>>>>>>>> <mailto:
> > > > > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is
> > an
> > > > > >> > >>> Airflow
> > > > > >> > >>>>>> term
> > > > > >> > >>>>>>>> now
> > > > > >> > >>>>>>>>>>>> rather
> > > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it
> or
> > > > not,
> > > > > >> > >>> I've
> > > > > >> > >>>>>>>> worked
> > > > > >> > >>>>>>>>> with
> > > > > >> > >>>>>>>>>>>> work
> > > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years
> > now
> > > > > >> > >>> and are
> > > > > >> > >>>>>> pro
> > > > > >> > >>>>>>>>> devs
> > > > > >> > >>>>>>>>>>> but
> > > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG!
> > For
> > > > > >> > >>> them,
> > > > > >> > >>>>> dag
> > > > > >> > >>>>>> =
> > > > > >> > >>>>>>>>> Airflow.
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms
> > but
> > > > > >> > >>> will be
> > > > > >> > >>>>> a
> > > > > >> > >>>>>>>> major
> > > > > >> > >>>>>>>>>>>> behavior
> > > > > >> > >>>>>>>>>>>> change for the developer community, so we'll
> need a
> > > > solid
> > > > > >> > >>>>> plan
> > > > > >> > >>>>>> on
> > > > > >> > >>>>>>>>> how to
> > > > > >> > >>>>>>>>>>>> introduce it, when we do.
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about
> the
> > > > Great
> > > > > >> > >>>>>> Renaming
> > > > > >> > >>>>>>>> in
> > > > > >> > >>>>>>>>> core
> > > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could
> > > probably
> > > > > >> > >>> get
> > > > > >> > >>>>>> some
> > > > > >> > >>>>>>>>>>> learnings
> > > > > >> > >>>>>>>>>>>> from there? Who knows!
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>> Guido's blog (2009):
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>
> > > > > >> > >>>>>>>
> > > > > >> > >>>>>
> > > > > >> > >>>
> > > > > >> >
> > > > > >>
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > >> > >>>>>>>>>>> <
> > > > > >> > >>>>>>>>>
> > > > > >> > >>>>>>>
> > > > > >> > >>>>>
> > > > > >> > >>>
> > > > > >> >
> > > > > >>
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > >> > >>>>>>>>>
> > > > > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>
> > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > >> > >>>>>>> <
> > > > > >> > >>>>>>>>>>>
> > > > > >> > >>>>>
> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > >
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>> Cheers,
> > > > > >> > >>>>>>>>>>>> Omkar
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > > > > >> > >>> weilee.rx@gmail.com
> > > > > >> > >>>>>>>>> <mailto:
> > > > > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an
> > > > airflow
> > > > > >> > >>>>> term.
> > > > > >> > >>>>>>> At
> > > > > >> > >>>>>>>>> least,
> > > > > >> > >>>>>>>>>>>>> thatâs how I understood it when I first used
> > > Airflow.
> > > > I
> > > > > >> > >>>>> feel
> > > > > >> > >>>>>>>>> renaming
> > > > > >> > >>>>>>>>>>> it
> > > > > >> > >>>>>>>>>>>> at
> > > > > >> > >>>>>>>>>>>>> this stage would require considerable effort
> from
> > > > > >> > >>>>> maintainers
> > > > > >> > >>>>>>> and
> > > > > >> > >>>>>>>>>>>> existing
> > > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > > > > >> > >>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>> Best,
> > > > > >> > >>>>>>>>>>>>> Wei
> > > > > >> > >>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > > > >> > >>>>>> kaxilnaik@gmail.com
> > > > > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > > > > >> > >>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we
> > > re-kindle
> > > > > >> > >>>>> this
> > > > > >> > >>>>>>>>> discussion
> > > > > >> > >>>>>>>>>>>> for
> > > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too
> > > much
> > > > > >> > >>>>>>> disruption
> > > > > >> > >>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance
> Martineau
> > > > > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > > > > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > > > > >> > >>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with
> > Airflow
> > > > > >> > >>>>>> experience
> > > > > >> > >>>>>>>>> what a
> > > > > >> > >>>>>>>>>>>> dag
> > > > > >> > >>>>>>>>>>>>> is,
> > > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow
> attributes
> > -
> > > > > >> > >>> stuff
> > > > > >> > >>>>>> like
> > > > > >> > >>>>>>>>> dags
> > > > > >> > >>>>>>>>>>>> being
> > > > > >> > >>>>>>>>>>>>> a
> > > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The
> > > structure
> > > > > >> > >>>>>> doesn't
> > > > > >> > >>>>>>>>> come up.
> > > > > >> > >>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> > > > > >> > >>> embrace
> > > > > >> > >>>>> the
> > > > > >> > >>>>>>> name
> > > > > >> > >>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure
> aspect.
> > > > > >> > >>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > > > > >> > >>>>> vikram@astronomer.io.inva
> > > > > >> > >>>>>>>>> lid>
> > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> > > > > >> > >>>>> struggling
> > > > > >> > >>>>>>> with
> > > > > >> > >>>>>>>>> this
> > > > > >> > >>>>>>>>>>>>> when I
> > > > > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it
> > being
> > > an
> > > > > >> > >>>>>>>> established
> > > > > >> > >>>>>>>>>>>> concept
> > > > > >> > >>>>>>>>>>>>>>> now
> > > > > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > > > > >> > >>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the
> perspective
> > > > > >> > >>> best
> > > > > >> > >>>>>>>> expressed
> > > > > >> > >>>>>>>>> by
> > > > > >> > >>>>>>>>>>>>> Daniel
> > > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word,
> > rather
> > > > > >> > >>> than
> > > > > >> > >>>>>> the
> > > > > >> > >>>>>>>>> computer
> > > > > >> > >>>>>>>>>>>>>>> science
> > > > > >> > >>>>>>>>>>>>>>>> concept.
> > > > > >> > >>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>> Best regards,
> > > > > >> > >>>>>>>>>>>>>>>> Vikram
> > > > > >> > >>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira,
> Niko
> > > > > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > > > > >> > >>>>>> onikolas@amazon.com.inva
> > > > > >> > >>>>>>>>> lid
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of:
> You're
> > > > > >> > >>> right
> > > > > >> > >>>>>> Ryan,
> > > > > >> > >>>>>>>> DAG
> > > > > >> > >>>>>>>>>>> isn't
> > > > > >> > >>>>>>>>>>>>>>> great
> > > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it
> will
> > > > > >> > >>> cause
> > > > > >> > >>>>> much
> > > > > >> > >>>>>>>> more
> > > > > >> > >>>>>>>>>>>> wreckage
> > > > > >> > >>>>>>>>>>>>>>>> than
> > > > > >> > >>>>>>>>>>>>>>>>> it solves.
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away
> > from
> > > > > >> > >>>>> defining
> > > > > >> > >>>>>>>> DAG.
> > > > > >> > >>>>>>>>> I
> > > > > >> > >>>>>>>>>>>> think
> > > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a
> community
> > > > > >> > >>> for a
> > > > > >> > >>>>>> while
> > > > > >> > >>>>>>>> now
> > > > > >> > >>>>>>>>>>>> anyway,
> > > > > >> > >>>>>>>>>>>>>>> so
> > > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> Cheers,
> > > > > >> > >>>>>>>>>>>>>>>>> Niko
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> ________________________________
> > > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org
> > > <mailto:
> > > > > >> > >>>>>>>>> ash@apache.org>>
> > > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > >> > >>>>>> dev@airflow.apache.org
> > > > > >> > >>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate
> > the
> > > > > >> > >>> term
> > > > > >> > >>>>>> "DAG"
> > > > > >> > >>>>>>>>> for end
> > > > > >> > >>>>>>>>>>>>>>> users
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside
> of
> > > the
> > > > > >> > >>>>>>>>> organization.
> > > > > >> > >>>>>>>>>>> Do
> > > > > >> > >>>>>>>>>>>>> not
> > > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you
> can
> > > > > >> > >>> confirm
> > > > > >> > >>>>>> the
> > > > > >> > >>>>>>>>> sender
> > > > > >> > >>>>>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>>> know
> > > > > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique
> > provient
> > > > > >> > >>> dâun
> > > > > >> > >>>>>>>>> expéditeur
> > > > > >> > >>>>>>>>>>>>>>> externe.
> > > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune
> > > piĂšce
> > > > > >> > >>>>>> jointe
> > > > > >> > >>>>>>> si
> > > > > >> > >>>>>>>>> vous
> > > > > >> > >>>>>>>>>>> ne
> > > > > >> > >>>>>>>>>>>>>>>> pouvez
> > > > > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et
> si
> > > > > >> > >>> vous
> > > > > >> > >>>>>> nâĂȘtes
> > > > > >> > >>>>>>>> pas
> > > > > >> > >>>>>>>>>>>> certain
> > > > > >> > >>>>>>>>>>>>>>>> que
> > > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ
> as a
> > > > > >> > >>> term â
> > > > > >> > >>>>>>>> getting
> > > > > >> > >>>>>>>>> to
> > > > > >> > >>>>>>>>>>>>> re-use
> > > > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg
> <
> > > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way,
> > > both
> > > > > >> > >>>>> sides
> > > > > >> > >>>>>>> have
> > > > > >> > >>>>>>>>> good
> > > > > >> > >>>>>>>>>>>>>>> reasons
> > > > > >> > >>>>>>>>>>>>>>>>> presented.
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>> -a
> > > > > >> > >>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA
> <mailto:
> > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name
> > comes
> > > > > >> > >>> from,
> > > > > >> > >>>>>>> "well,
> > > > > >> > >>>>>>>>> no
> > > > > >> > >>>>>>>>>>> one
> > > > > >> > >>>>>>>>>>>>>>>>> actually
> > > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek
> Potiuk
> > <
> > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph"
> > is
> > > > > >> > >>> the
> > > > > >> > >>>>>>>>>>> tongue-twister,
> > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek
> > Potiuk <
> > > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote.
> I
> > > > > >> > >>> would
> > > > > >> > >>>>> very
> > > > > >> > >>>>>>>> much
> > > > > >> > >>>>>>>>> want
> > > > > >> > >>>>>>>>>>>> to
> > > > > >> > >>>>>>>>>>>>>>> be
> > > > > >> > >>>>>>>>>>>>>>>>>>> able
> > > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> > > > > >> > >>> further.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a
> talk
> > > > > >> > >>> it's a
> > > > > >> > >>>>>>>>>>>> tongue-twister,
> > > > > >> > >>>>>>>>>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>>>> I
> > > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> > > > > >> > >>> pronounce
> > > > > >> > >>>>>> it
> > > > > >> > >>>>>>>>>>> properly.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>> J.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent
> > Bovenzi
> > > > > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > > > > >> > >>>>>>>> brent@astronomer.io.inva
> > > > > >> > >>>>>>>>>> lid>
> > > > > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG"
> when
> > > > > >> > >>>>> starting
> > > > > >> > >>>>>> on
> > > > > >> > >>>>>>>>> AIP-38
> > > > > >> > >>>>>>>>>>> to
> > > > > >> > >>>>>>>>>>>>>>>>>>> modernize
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow"
> are
> > > more
> > > > > >> > >>>>>>>>> descriptive of
> > > > > >> > >>>>>>>>>>>> what
> > > > > >> > >>>>>>>>>>>>>>>> one
> > > > > >> > >>>>>>>>>>>>>>>>>>> is
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic
> > Graph
> > > > > >> > >>> is an
> > > > > >> > >>>>>>>>>>> implementation
> > > > > >> > >>>>>>>>>>>>>>>>> detail.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at
> this
> > > > > >> > >>> point
> > > > > >> > >>>>>> "DAG"
> > > > > >> > >>>>>>>> has
> > > > > >> > >>>>>>>>>>> become
> > > > > >> > >>>>>>>>>>>>>>>> "dag"
> > > > > >> > >>>>>>>>>>>>>>>>>>> , a
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in
> community
> > > > > >> > >>>>>> discussion,
> > > > > >> > >>>>>>>>> Airflow
> > > > > >> > >>>>>>>>>>>>>>>> Summit
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI.
> > > Let's
> > > > > >> > >>>>>> embrace
> > > > > >> > >>>>>>>>> "dag".
> > > > > >> > >>>>>>>>>>> A
> > > > > >> > >>>>>>>>>>>>>>> user
> > > > > >> > >>>>>>>>>>>>>>>>>>> just
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the
> > technical
> > > > > >> > >>>>> concept
> > > > > >> > >>>>>>>>> behind
> > > > > >> > >>>>>>>>>>> that
> > > > > >> > >>>>>>>>>>>>>>>>> word. I
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > > > > >> > >>> refactoring so
> > > > > >> > >>>>>>> much
> > > > > >> > >>>>>>>>> code,
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack
> overflow
> > > > > >> > >>>>> questions,
> > > > > >> > >>>>>>>> etc.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> > > > > >> > >>> Standish
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva
> > > <mailto:
> > > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing
> a
> > > lot
> > > > > >> > >>> of
> > > > > >> > >>>>>> pain
> > > > > >> > >>>>>>>> for
> > > > > >> > >>>>>>>>>>>>>>>>> questionable
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to
> the
> > > > > >> > >>> idea. I
> > > > > >> > >>>>>>> agree
> > > > > >> > >>>>>>>>> the
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> association
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not
> > > helpful.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here
> some
> > > less
> > > > > >> > >>>>>>> invasive
> > > > > >> > >>>>>>>>>>>>>>> mitigations.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is
> to
> > > > > >> > >>>>>> de-emphasize
> > > > > >> > >>>>>>>> the
> > > > > >> > >>>>>>>>>>> math
> > > > > >> > >>>>>>>>>>>>>>> nerd
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> > > > > >> > >>> website /
> > > > > >> > >>>>>> etc,
> > > > > >> > >>>>>>>>> *never
> > > > > >> > >>>>>>>>>>>>>>>> define*
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed
> > > acyclic
> > > > > >> > >>>>>> graph.
> > > > > >> > >>>>>>>>> Always
> > > > > >> > >>>>>>>>>>>>>>> define
> > > > > >> > >>>>>>>>>>>>>>>>> it
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow
> > > etc.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it
> > is
> > > > > >> > >>> like a
> > > > > >> > >>>>>>>>> historical
> > > > > >> > >>>>>>>>>>>>>>>>> footnote,
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it
> somewhere
> > > > > >> > >>> hidden.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in
> > the
> > > > > >> > >>> docs
> > > > > >> > >>>>> in
> > > > > >> > >>>>>>>>> general
> > > > > >> > >>>>>>>>>>> e.g.
> > > > > >> > >>>>>>>>>>>>>>>>>>> writing
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing
> "DAG" /
> > > > > >> > >>> "DAGs"
> > > > > >> > >>>>>> etc.
> > > > > >> > >>>>>>>> The
> > > > > >> > >>>>>>>>>>>> upper
> > > > > >> > >>>>>>>>>>>>>>>> case
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> part
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym;
> but
> > > > > >> > >>> "dag" in
> > > > > >> > >>>>>>>>> airlfow is
> > > > > >> > >>>>>>>>>>>> just
> > > > > >> > >>>>>>>>>>>>>>>> an
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association
> with
> > > > > >> > >>> "DAGs"
> > > > > >> > >>>>> is
> > > > > >> > >>>>>>> not
> > > > > >> > >>>>>>>>> really
> > > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in
> > > airflow
> > > > > >> > >>> is
> > > > > >> > >>>>> its
> > > > > >> > >>>>>>> own
> > > > > >> > >>>>>>>>>>> thing,
> > > > > >> > >>>>>>>>>>>> is
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph
> (which
> > > > > >> > >>> nobody
> > > > > >> > >>>>>> knows
> > > > > >> > >>>>>>>>> about
> > > > > >> > >>>>>>>>>>>>>>>> anyway),
> > > > > >> > >>>>>>>>>>>>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms
> that
> > > > > >> > >>> normal
> > > > > >> > >>>>>>> people
> > > > > >> > >>>>>>>>>>>>>>> understand.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek
> > > Potiuk
> > > > > >> > >>> <
> > > > > >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > > > > >> > >>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do
> that
> > it
> > > > > >> > >>> will
> > > > > >> > >>>>> be
> > > > > >> > >>>>>>>>> extremely
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think
> > it
> > > > > >> > >>> will
> > > > > >> > >>>>>> make
> > > > > >> > >>>>>>> a
> > > > > >> > >>>>>>>>> lot of
> > > > > >> > >>>>>>>>>>>>>>>>>>> "google"
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches
> > not
> > > > > >> > >>>>> finding
> > > > > >> > >>>>>>> the
> > > > > >> > >>>>>>>>> right
> > > > > >> > >>>>>>>>>>>>>>>>>>> answers.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> This
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> > > > > >> > >>> besides the
> > > > > >> > >>>>>>>>> community
> > > > > >> > >>>>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>>> ideas
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> that
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number
> of
> > > > > >> > >>>>> examples,
> > > > > >> > >>>>>>>>> problems
> > > > > >> > >>>>>>>>>>> and
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> > > > > >> > >>> remember
> > > > > >> > >>>>>> that
> > > > > >> > >>>>>>>>> all the
> > > > > >> > >>>>>>>>>>>> AI
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> trained
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> on
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> > > > > >> > >>> matching
> > > > > >> > >>>>>>> queries
> > > > > >> > >>>>>>>> of
> > > > > >> > >>>>>>>>>>>> people.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could
> > > easily
> > > > > >> > >>>>>> switch.
> > > > > >> > >>>>>>>>> And if
> > > > > >> > >>>>>>>>>>> we
> > > > > >> > >>>>>>>>>>>>>>> do
> > > > > >> > >>>>>>>>>>>>>>>> -
> > > > > >> > >>>>>>>>>>>>>>>>>>> I
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or
> pipeline
> > > > > >> > >>> instead
> > > > > >> > >>>>> of
> > > > > >> > >>>>>>>>> `dag` if
> > > > > >> > >>>>>>>>>>>> not
> > > > > >> > >>>>>>>>>>>>>>>> the
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> above
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with
> Igor
> > > > > >> > >>> that it
> > > > > >> > >>>>>>> might
> > > > > >> > >>>>>>>>> cause
> > > > > >> > >>>>>>>>>>>> more
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> problems
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others
> > will
> > > > > >> > >>> think
> > > > > >> > >>>>>>> it's
> > > > > >> > >>>>>>>> a
> > > > > >> > >>>>>>>>> good
> > > > > >> > >>>>>>>>>>>>>>>> idea, I
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> am
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM
> Abhishek
> > > > > >> > >>> Bhakat
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> > > > > >> > >>> <mailto:
> > > > > >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very
> > less
> > > > > >> > >>> sense
> > > > > >> > >>>>> to
> > > > > >> > >>>>>>>>> someone
> > > > > >> > >>>>>>>>>>> new
> > > > > >> > >>>>>>>>>>>>>>> to
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show
> > the
> > > > > >> > >>> nature
> > > > > >> > >>>>>> of
> > > > > >> > >>>>>>>>> being
> > > > > >> > >>>>>>>>>>>>>>> acyclic.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> Sure,
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> as
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> > > > > >> > >>> workaround it.
> > > > > >> > >>>>>>>> Still,
> > > > > >> > >>>>>>>>> in
> > > > > >> > >>>>>>>>>>> my
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> opinion,
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> there
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic
> > behavior
> > > in
> > > > > >> > >>>>>>> workflow
> > > > > >> > >>>>>>>>>>>>>>>>>>> orchestration.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> Most
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way
> can
> > be
> > > > > >> > >>>>> covered
> > > > > >> > >>>>>>>>> using an
> > > > > >> > >>>>>>>>>>>>>>>> acyclic
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> manner
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> > > > > >> > >>> idempotency. So
> > > > > >> > >>>>> I
> > > > > >> > >>>>>>>> would
> > > > > >> > >>>>>>>>> want
> > > > > >> > >>>>>>>>>>>> the
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > > >> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> > > > > >> > >>> bernd.stroehle@kosakya.de>>
> > > > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become
> > an
> > > > > >> > >>>>> Airflow
> > > > > >> > >>>>>>> Fan;
> > > > > >> > >>>>>>>>> so
> > > > > >> > >>>>>>>>>>> many
> > > > > >> > >>>>>>>>>>>>>>> new
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it
> should
> > be
> > > > > >> > >>>>>> replaced
> > > > > >> > >>>>>>> by
> > > > > >> > >>>>>>>>> the
> > > > > >> > >>>>>>>>>>>> more
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> general
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or
> > > Airflow
> > > > > >> > >>>>>> (Petri)
> > > > > >> > >>>>>>>> Net
> > > > > >> > >>>>>>>>>>> (AFN)
> > > > > >> > >>>>>>>>>>>>>>>>>>> (maybe
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be
> stored
> > > in a
> > > > > >> > >>>>> Graph
> > > > > >> > >>>>>>>>> Database.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an
> Airflow
> > > > > >> > >>> Graph
> > > > > >> > >>>>>> (AFG)
> > > > > >> > >>>>>>>>> might be
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> assigned
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>> to
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>> an
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... )
> > member
> > > > > >> > >>> of a
> > > > > >> > >>>>>>>> library.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a
> different
> > > > > >> > >>>>> structure
> > > > > >> > >>>>>>>> than
> > > > > >> > >>>>>>>>> a
> > > > > >> > >>>>>>>>>>>>>>>>>>>>> configuration
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's
> > bullshit.
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com
> <mailto:
> > > > > >> > >>>>>>>>>>> bernd.stroehle@gmail.com>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> > > > > >> > >>> <ikholopov@google.com.INVA
> > > > > >> > >>>>>>>>> <mailto:
> > > > > >> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024
> 12:02
> > PM
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > >> > >>>>>>>>> dev@airflow.apache.org
> > > > > >> > >>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should
> deprecate
> > > the
> > > > > >> > >>> term
> > > > > >> > >>>>>>> "DAG"
> > > > > >> > >>>>>>>>> for
> > > > > >> > >>>>>>>>>>> end
> > > > > >> > >>>>>>>>>>>>>>>>>>> users
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is
> clearly
> > > > > >> > >>>>>> suboptimal,
> > > > > >> > >>>>>>> it
> > > > > >> > >>>>>>>>> is
> > > > > >> > >>>>>>>>>>> part
> > > > > >> > >>>>>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Ambika Garg <am...@gmail.com>.
+1 would vote for
Airflow's *D*ynamic *A*ction *G*raph (DAG)
*Dynamic*: Reflects the capability of Airflow to handle workflows that
arenât strictly acyclic, but can adapt to changes, iterations and loops
within broader context of workflow.
*Action*: Emphasizes that airflow is fundamentally about orchestrating
actions or tasks. Whether its data processing, model training or any other
form of computation. It also conveys operational aspect of what Airflow
does.
*Graph*: Maintains the core concept that workflows in Airflow are
visualized and managed as graphs. This term keeps the idea of dependencies
and relationships between tasks intact.
Ambika.
On Tue, Feb 18, 2025 at 11:46âŻAM Brent Bovenzi <br...@astronomer.io.invalid>
wrote:
> -0.5
>
> Even though the new UI prefers "Dag", I don't think that means "DAG" is
> incorrect.
>
> On Tue, Feb 18, 2025 at 11:36âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I love "Dynamic Airflow Graph" +10 to that Constance.
> >
> > And yes. I fully agree with your "somewhat" acyclic statement. I saw
> people
> > mentioning "event-driven" workflows to better describe the AI/ML, but
> that
> > for me is a synonym of "completely unmanageable workflows" (similar to
> > micro-services) - what I was alluding to was exactly this - something
> like
> > curren DAG with parts of it being potentially cyclic - so we are
> completely
> > aligned here :). And that's what make it "not DAG - in mathematical
> sense"
> > - even more reason to depart from it,
> >
> > J.
> >
> > On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
> > <co...@astronomer.io.invalid> wrote:
> >
> > > Hey all,
> > >
> > > I appreciate the discussion about how we position the concept of a DAG
> in
> > > Airflow. While I agree with updating documentation to de-emphasize the
> > > strict mathematical definition, I do not think we should change the
> > public
> > > interface to use "dag" instead of "DAG" everywhere. It adds
> > > unnecessary churn and breaks consistency across the ecosystem.
> > >
> > > Regarding the broader discussion on whether DAGs in Airflow should
> remain
> > > acyclic: I acknowledge that some workflows, particularly in ML and AI,
> > > require cyclical behaviour (example: iterative refinement until a
> > condition
> > > is met). However, I think it's important to recognize that even in
> those
> > > cases, the workflow itself still has an overall start and end. It's not
> > an
> > > unbounded loop but rather a controlled execution process that happens
> to
> > > have cycles *within* it.
> > >
> > > That being said, I do see value in reinterpreting "DAG" to better align
> > > with how Airflow is actually used, and love shifting away from
> "Directed
> > > Acyclic Graph" in favour of something more intuitive. However "Data
> > Airflow
> > > Graph" makes it sound like we're defining Airflow itself rather than
> > > describing the structure of workflows within it.
> > >
> > > If we want a new meaning for DAG that resonates with our base while
> > keeping
> > > Airflow's identity clear, I am proposing three alternatives:
> > >
> > >
> > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while
> emphasizing
> > > flexibility
> > > 2. *Distributed Automation Graph*: Highlights execution across
> systems
> > > and automations without tying it strictly to data
> > > 3. *Directed Activity Graph*: Retains the useful meaning of
> "Directed"
> > > while shifting away from strict acyclic constraints. (My personal
> > > favourite)
> > >
> > > I think any shift in terminology should be driven by what best
> > communicates
> > > Airflow's role workflow automation, rather than just distancing
> ourselves
> > > from the mathematical definition.
> > >
> > > Curious to hear more thoughts!
> > >
> > > On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > > > Sorry for the few typos ... My slight dyslexia did not help and
> > > autocorrect
> > > > did not help either this time : dept -> debt, routed -> rooted and a
> > few
> > > > other small typos :) .
> > > >
> > > > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > First of all - yes, agree with Lee and Ash, I think we do not need
> to
> > > > > change the public interface, I'd also be -0 on that, but docs,
> > internal
> > > > > strings, yes we could.
> > > > >
> > > > > But .... After thinking a bit and looking at the discussion here,
> > where
> > > > > people have concerns, I have an alternative take. And I know this
> > might
> > > > be
> > > > > controversial, and some people might find it confusing, but why
> don't
> > > we
> > > > > come up with a new acronym for DAG, one that will be more airflow
> and
> > > > less
> > > > > "mathematics/algorithm". And I have, I think a good reason for it,
> > and
> > > I
> > > > > even have a proposal. Bear with me,
> > > > >
> > > > > DAG => *Data Airflow Graph*
> > > > >
> > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now
> > (with
> > > > > exception of some niche products - you know which one I speak
> about -
> > > > which
> > > > > are unfortunate enough to use the "Dag" in their name - I will
> > explain
> > > > why
> > > > > this is unfortunate, IMHO).
> > > > >
> > > > > We are lucky to have Airflow for "A" in the acronym and we can make
> > it
> > > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow.
> > > > >
> > > > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > > > >
> > > > > Not only because "Directed" and "Acyclic" are mostly not in the
> > > > vocabulary
> > > > > of most of our users - as Ryan, starting this thread explained
> yes, I
> > > > agree
> > > > > with it, but also I think there is a more important reason - I
> > simply
> > > > > believe in the future our DAGs might not (and likely will not) be
> > > > "acyclic".
> > > > >
> > > > > I know for some it might be herezy and we have not discussed any of
> > > that
> > > > > yet, but more and more I keep on hearing that Machine Learning
> > > Workflows
> > > > of
> > > > > the future are often somewhat cyclic. In our case there is one type
> > of
> > > > > repeat-in-cycle behaviour - we have "task retry on failure" - but
> > this
> > > is
> > > > > only on failure and it basically mens "idempotent" retry - we want
> to
> > > > redo
> > > > > something and get single result out of that and basically discard
> the
> > > > > previous result. But machine learning/AI workflows are different. I
> > > had a
> > > > > lot of conversations recently (I even co-organized "AI low-level
> > > > > engineering and hacking" devroom to learn more about those) - and I
> > > keep
> > > > on
> > > > > hearing that "Repeat a sequence of things several times and refine"
> > is
> > > > > precisely what AI / ML workflows will need - especially with
> Agentic
> > > > > workflows - often we have cooperating agents which **could** be
> part
> > of
> > > > the
> > > > > same DAG, but currently in Airflow where our DAGs are "Acyclic" we
> > can
> > > > > only express "repeat several time and refine" as a single task that
> > > will
> > > > do
> > > > > such refining. If we would like to split such an "agent" doing
> their
> > > own
> > > > > work in "cycles" and communicating with other "agents" doing the
> > same -
> > > > > represented as a multi-task-entity, we have no way to express it
> > using
> > > > our
> > > > > current "acyclic" semantics.
> > > > >
> > > > > But - other than our implementation makes that assumption
> currently -
> > > > > there is nothing fundamental with Airflow being able to run such
> > > > "partially
> > > > > cyclic workflows". We have all the pieces, we would have to change
> > > > > dependency calculation to allow for cycles, update UI to handle
> > cycles
> > > > > representation - and we could in theory quite seamlessly, support
> > > > "cyclic"
> > > > > workflows.
> > > > >
> > > > > I know, I know, that's the first time we hear it and that's huge
> > > change -
> > > > > of course, I do not want to start discussing any details of it, not
> > > > before
> > > > > Airflow 3.0 is out - but I think we should be open that at some
> point
> > > of
> > > > > time the "Acyclic" property is going to go away, And my personal
> > > > intuition
> > > > > tells me that a year from now we will have acyclic workflows
> > (assuming
> > > > that
> > > > > all the foundational work we do on Airflow 3 will really help in
> > > > decreasing
> > > > > all the technical dept, and will open our minds to new ideas).
> > > > >
> > > > > That's why I think - the sooner we get rid of "Directed Acyclic" -
> > the
> > > > > better. that will - even for us, maintainer free us from being
> routed
> > > in
> > > > > the acyclic nature of Airflow.
> > > > >
> > > > > How about we change to "Data Airflow Graph" (or any other good
> > acronym
> > > > > which has (A)irflow instead of (A)cyclic). The name is just a
> > proposal.
> > > > >
> > > > > Now.. I go hide somewhere as I believe that might cause a
> "flamewar"
> > > :),
> > > > > unless on request of Ryan you already sheathed your swords in the
> > > > > discussion (which seems everyone did).
> > > > >
> > > > > But honestly - I would really love what you think of "let's just
> > break
> > > > > with Acyclic NOW" and shape our future by using a different
> acronym.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > > > ephraimanierobi@gmail.com> wrote:
> > > > >
> > > > >> When I started contributing to Airflow, I had to read up on DAG,
> and
> > > > now I
> > > > >> know what it means. I wonder if we are about to have users who use
> > the
> > > > >> term
> > > > >> Dag without knowing that there's a DAG.
> > > > >>
> > > > >> Slightly concerned.
> > > > >>
> > > > >> -ephraim
> > > > >>
> > > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com>
> wrote:
> > > > >>
> > > > >> > For user-facing things (e.g., `from airflow import DAG`), Iâm
> more
> > > > like
> > > > >> > -0. But for documentation, docstring, internal things and etc.,
> we
> > > > >> probably
> > > > >> > could still change most of them?
> > > > >> >
> > > > >> > Best,
> > > > >> > Wei
> > > > >> >
> > > > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <
> ash@apache.org>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > I also donât personally think itâs worth the pain (not to
> > mention
> > > > >> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on
> > that.
> > > > >> > >
> > > > >> > > -ash
> > > > >> > >
> > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > > > daniel.imberman@gmail.com
> > > > >> >
> > > > >> > wrote:
> > > > >> > >>
> > > > >> > >> I think my biggest concern is a marketing one and not a
> > technical
> > > > >> one.
> > > > >> > >>
> > > > >> > >> As has been mentioned on the thread the terms airflow and dag
> > are
> > > > >> kind
> > > > >> > of
> > > > >> > >> synonymous and I certainly donât want to give the impression
> > that
> > > > we
> > > > >> are
> > > > >> > >> breaking more than we are breaking.
> > > > >> > >>
> > > > >> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> > > > >> > >>
> > > > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org>
> > > wrote:
> > > > >> > >>
> > > > >> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff
> > rules
> > > > are
> > > > >> > best
> > > > >> > >>> suited for user-facing things but not the airflow code base
> > > > itself.
> > > > >> If
> > > > >> > what
> > > > >> > >>> we mean is adding a rule to avoid users using "DAG" *after*
> we
> > > > >> rename
> > > > >> > it,
> > > > >> > >>> it's definitely a +1000.
> > > > >> > >>>
> > > > >> > >>> I just created GitHub issues for this removing "DAG"
> > > > >> > >>>
> > > > >> > >>> https://github.com/apache/airflow/issues/46842
> > > > >> > >>> https://github.com/apache/airflow/issues/46843
> > > > >> > >>>
> > > > >> > >>> One thing I'm not sure about is whether we want to get rid
> of
> > > > `from
> > > > >> > >>> airflow import DAG` as well. đ€
> > > > >> > >>>
> > > > >> > >>> Best,
> > > > >> > >>> Wei
> > > > >> > >>>
> > > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > > > >> > >>>> Hard to say until it's looked at :)
> > > > >> > >>>>
> > > > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > > > >> aritrabasu1999@gmail.com
> > > > >> > >
> > > > >> > >>>> wrote:
> > > > >> > >>>>
> > > > >> > >>>>> I can take it up, it's mostly just a doc update right? Or
> > are
> > > we
> > > > >> > doing
> > > > >> > >>> code
> > > > >> > >>>>> files replacement too?
> > > > >> > >>>>> --
> > > > >> > >>>>> Regards,
> > > > >> > >>>>> Aritra Basu
> > > > >> > >>>>>
> > > > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <
> > jarek@potiuk.com
> > > >
> > > > >> > wrote:
> > > > >> > >>>>>
> > > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any
> > > > >> volunteers
> > > > >> > >>> to
> > > > >> > >>>>>> make it ? It's fun.
> > > > >> > >>>>>>
> > > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > > > droiddev510e@gmail.com>
> > > > >> > >>> wrote:
> > > > >> > >>>>>>
> > > > >> > >>>>>>> +1 for ruff rules :)
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG'
> in
> > > the
> > > > >> > >>> Airflow
> > > > >> > >>>>>>> docs, in line with the new UI changes and to make the
> > > renaming
> > > > >> > >>>>>>> consistent across user-facing pages.
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> Regards,
> > > > >> > >>>>>>> Omkar
> > > > >> > >>>>>>>
> > > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > > > jarek@potiuk.com>
> > > > >> > >>> wrote:
> > > > >> > >>>>>>>
> > > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <
> > weilee@apache.org
> > > >
> > > > >> > >>> wrote:
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag"
> or
> > > > "Dag"
> > > > >> > >>>>> instead
> > > > >> > >>>>>>> of
> > > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG"
> in
> > > the
> > > > >> > >>>>> codebase
> > > > >> > >>>>>>> with
> > > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it
> > might
> > > be
> > > > >> > >>> đ€),
> > > > >> > >>>>>>> should
> > > > >> > >>>>>>>> we
> > > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the following
> > PRs?
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> Best,
> > > > >> > >>>>>>>>> Wei
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > > > >> > >>>>>>>>>> Let me know if that's the direction we want to go.
> > > > >> > >>>>>>>>>>
> > > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > > > >> > >>>>>>>>>>
> > > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
> > > > >> > >>> synonymous
> > > > >> > >>>>>> with
> > > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and
> > explain
> > > > >> > >>> what
> > > > >> > >>>>>>> Airflow
> > > > >> > >>>>>>>>> is
> > > > >> > >>>>>>>>>>> all about. Although folks would ask, I donât ever
> use
> > > the
> > > > >> > >>>>>>>> mathematical
> > > > >> > >>>>>>>>>>> definition of DAG.
> > > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for
> many
> > > > >> > >>> users
> > > > >> > >>>>>> making
> > > > >> > >>>>>>>>> such a
> > > > >> > >>>>>>>>>>> change - I would rather direct that energy to
> > appending
> > > > the
> > > > >> > >>>>>> Oxford
> > > > >> > >>>>>>>>>>> definition of "dag" to include a reference to
> > workflows
> > > in
> > > > >> > >>>>>> Airflow.
> > > > >> > >>>>>>>> __
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> -- Rajesh
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> > > jarek@potiuk.com
> > > > >> > >>>>>> <mailto:
> > > > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the
> > > > >> > >>>>> organization.
> > > > >> > >>>>>> Do
> > > > >> > >>>>>>>> not
> > > > >> > >>>>>>>>>>> click links or open attachments unless you can
> confirm
> > > the
> > > > >> > >>>>> sender
> > > > >> > >>>>>>> and
> > > > >> > >>>>>>>>> know
> > > > >> > >>>>>>>>>>> the content is safe.
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> dâun
> > > > >> > >>>>> expéditeur
> > > > >> > >>>>>>>>> externe.
> > > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > > jointe
> > > > >> > >>> si
> > > > >> > >>>>> vous
> > > > >> > >>>>>>> ne
> > > > >> > >>>>>>>>> pouvez
> > > > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous
> > > nâĂȘtes
> > > > >> > >>> pas
> > > > >> > >>>>>>>> certain
> > > > >> > >>>>>>>>> que
> > > > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> From Guido's post:
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been
> > > > renamed".
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> Naming is hard.
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> J.
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > > >> > >>>>> droiddev510e@gmail.com
> > > > >> > >>>>>>>>> <mailto:
> > > > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is
> an
> > > > >> > >>> Airflow
> > > > >> > >>>>>> term
> > > > >> > >>>>>>>> now
> > > > >> > >>>>>>>>>>>> rather
> > > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or
> > > not,
> > > > >> > >>> I've
> > > > >> > >>>>>>>> worked
> > > > >> > >>>>>>>>> with
> > > > >> > >>>>>>>>>>>> work
> > > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years
> now
> > > > >> > >>> and are
> > > > >> > >>>>>> pro
> > > > >> > >>>>>>>>> devs
> > > > >> > >>>>>>>>>>> but
> > > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG!
> For
> > > > >> > >>> them,
> > > > >> > >>>>> dag
> > > > >> > >>>>>> =
> > > > >> > >>>>>>>>> Airflow.
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms
> but
> > > > >> > >>> will be
> > > > >> > >>>>> a
> > > > >> > >>>>>>>> major
> > > > >> > >>>>>>>>>>>> behavior
> > > > >> > >>>>>>>>>>>> change for the developer community, so we'll need a
> > > solid
> > > > >> > >>>>> plan
> > > > >> > >>>>>> on
> > > > >> > >>>>>>>>> how to
> > > > >> > >>>>>>>>>>>> introduce it, when we do.
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the
> > > Great
> > > > >> > >>>>>> Renaming
> > > > >> > >>>>>>>> in
> > > > >> > >>>>>>>>> core
> > > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could
> > probably
> > > > >> > >>> get
> > > > >> > >>>>>> some
> > > > >> > >>>>>>>>>>> learnings
> > > > >> > >>>>>>>>>>>> from there? Who knows!
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>> Guido's blog (2009):
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>
> > > > >> > >>>
> > > > >> >
> > > > >>
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > >> > >>>>>>>>>>> <
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>
> > > > >> > >>>>>
> > > > >> > >>>
> > > > >> >
> > > > >>
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > >> > >>>>>>>>>
> > > > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>
> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > >> > >>>>>>> <
> > > > >> > >>>>>>>>>>>
> > > > >> > >>>>>
> > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > >
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>> Cheers,
> > > > >> > >>>>>>>>>>>> Omkar
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > > > >> > >>> weilee.rx@gmail.com
> > > > >> > >>>>>>>>> <mailto:
> > > > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an
> > > airflow
> > > > >> > >>>>> term.
> > > > >> > >>>>>>> At
> > > > >> > >>>>>>>>> least,
> > > > >> > >>>>>>>>>>>>> thatâs how I understood it when I first used
> > Airflow.
> > > I
> > > > >> > >>>>> feel
> > > > >> > >>>>>>>>> renaming
> > > > >> > >>>>>>>>>>> it
> > > > >> > >>>>>>>>>>>> at
> > > > >> > >>>>>>>>>>>>> this stage would require considerable effort from
> > > > >> > >>>>> maintainers
> > > > >> > >>>>>>> and
> > > > >> > >>>>>>>>>>>> existing
> > > > >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > > > >> > >>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>> Best,
> > > > >> > >>>>>>>>>>>>> Wei
> > > > >> > >>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > > >> > >>>>>> kaxilnaik@gmail.com
> > > > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > > > >> > >>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we
> > re-kindle
> > > > >> > >>>>> this
> > > > >> > >>>>>>>>> discussion
> > > > >> > >>>>>>>>>>>> for
> > > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too
> > much
> > > > >> > >>>>>>> disruption
> > > > >> > >>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > > > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > > > >> > >>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with
> Airflow
> > > > >> > >>>>>> experience
> > > > >> > >>>>>>>>> what a
> > > > >> > >>>>>>>>>>>> dag
> > > > >> > >>>>>>>>>>>>> is,
> > > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes
> -
> > > > >> > >>> stuff
> > > > >> > >>>>>> like
> > > > >> > >>>>>>>>> dags
> > > > >> > >>>>>>>>>>>> being
> > > > >> > >>>>>>>>>>>>> a
> > > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The
> > structure
> > > > >> > >>>>>> doesn't
> > > > >> > >>>>>>>>> come up.
> > > > >> > >>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> > > > >> > >>> embrace
> > > > >> > >>>>> the
> > > > >> > >>>>>>> name
> > > > >> > >>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
> > > > >> > >>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > > > >> > >>>>> vikram@astronomer.io.inva
> > > > >> > >>>>>>>>> lid>
> > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> > > > >> > >>>>> struggling
> > > > >> > >>>>>>> with
> > > > >> > >>>>>>>>> this
> > > > >> > >>>>>>>>>>>>> when I
> > > > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it
> being
> > an
> > > > >> > >>>>>>>> established
> > > > >> > >>>>>>>>>>>> concept
> > > > >> > >>>>>>>>>>>>>>> now
> > > > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > > > >> > >>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
> > > > >> > >>> best
> > > > >> > >>>>>>>> expressed
> > > > >> > >>>>>>>>> by
> > > > >> > >>>>>>>>>>>>> Daniel
> > > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word,
> rather
> > > > >> > >>> than
> > > > >> > >>>>>> the
> > > > >> > >>>>>>>>> computer
> > > > >> > >>>>>>>>>>>>>>> science
> > > > >> > >>>>>>>>>>>>>>>> concept.
> > > > >> > >>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>> Best regards,
> > > > >> > >>>>>>>>>>>>>>>> Vikram
> > > > >> > >>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > > > >> > >>>>>> onikolas@amazon.com.inva
> > > > >> > >>>>>>>>> lid
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
> > > > >> > >>> right
> > > > >> > >>>>>> Ryan,
> > > > >> > >>>>>>>> DAG
> > > > >> > >>>>>>>>>>> isn't
> > > > >> > >>>>>>>>>>>>>>> great
> > > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
> > > > >> > >>> cause
> > > > >> > >>>>> much
> > > > >> > >>>>>>>> more
> > > > >> > >>>>>>>>>>>> wreckage
> > > > >> > >>>>>>>>>>>>>>>> than
> > > > >> > >>>>>>>>>>>>>>>>> it solves.
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away
> from
> > > > >> > >>>>> defining
> > > > >> > >>>>>>>> DAG.
> > > > >> > >>>>>>>>> I
> > > > >> > >>>>>>>>>>>> think
> > > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
> > > > >> > >>> for a
> > > > >> > >>>>>> while
> > > > >> > >>>>>>>> now
> > > > >> > >>>>>>>>>>>> anyway,
> > > > >> > >>>>>>>>>>>>>>> so
> > > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> Cheers,
> > > > >> > >>>>>>>>>>>>>>>>> Niko
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> ________________________________
> > > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org
> > <mailto:
> > > > >> > >>>>>>>>> ash@apache.org>>
> > > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > >> > >>>>>> dev@airflow.apache.org
> > > > >> > >>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate
> the
> > > > >> > >>> term
> > > > >> > >>>>>> "DAG"
> > > > >> > >>>>>>>>> for end
> > > > >> > >>>>>>>>>>>>>>> users
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of
> > the
> > > > >> > >>>>>>>>> organization.
> > > > >> > >>>>>>>>>>> Do
> > > > >> > >>>>>>>>>>>>> not
> > > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can
> > > > >> > >>> confirm
> > > > >> > >>>>>> the
> > > > >> > >>>>>>>>> sender
> > > > >> > >>>>>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>>> know
> > > > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique
> provient
> > > > >> > >>> dâun
> > > > >> > >>>>>>>>> expéditeur
> > > > >> > >>>>>>>>>>>>>>> externe.
> > > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune
> > piĂšce
> > > > >> > >>>>>> jointe
> > > > >> > >>>>>>> si
> > > > >> > >>>>>>>>> vous
> > > > >> > >>>>>>>>>>> ne
> > > > >> > >>>>>>>>>>>>>>>> pouvez
> > > > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> > > > >> > >>> vous
> > > > >> > >>>>>> nâĂȘtes
> > > > >> > >>>>>>>> pas
> > > > >> > >>>>>>>>>>>> certain
> > > > >> > >>>>>>>>>>>>>>>> que
> > > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
> > > > >> > >>> term â
> > > > >> > >>>>>>>> getting
> > > > >> > >>>>>>>>> to
> > > > >> > >>>>>>>>>>>>> re-use
> > > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way,
> > both
> > > > >> > >>>>> sides
> > > > >> > >>>>>>> have
> > > > >> > >>>>>>>>> good
> > > > >> > >>>>>>>>>>>>>>> reasons
> > > > >> > >>>>>>>>>>>>>>>>> presented.
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>> -a
> > > > >> > >>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name
> comes
> > > > >> > >>> from,
> > > > >> > >>>>>>> "well,
> > > > >> > >>>>>>>>> no
> > > > >> > >>>>>>>>>>> one
> > > > >> > >>>>>>>>>>>>>>>>> actually
> > > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> > > > >> > >>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk
> <
> > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > >> > >>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph"
> is
> > > > >> > >>> the
> > > > >> > >>>>>>>>>>> tongue-twister,
> > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek
> Potiuk <
> > > > >> > >>>>>>>>> jarek@potiuk.com
> > > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > > >> > >>>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
> > > > >> > >>> would
> > > > >> > >>>>> very
> > > > >> > >>>>>>>> much
> > > > >> > >>>>>>>>> want
> > > > >> > >>>>>>>>>>>> to
> > > > >> > >>>>>>>>>>>>>>> be
> > > > >> > >>>>>>>>>>>>>>>>>>> able
> > > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> > > > >> > >>> further.
> > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
> > > > >> > >>> it's a
> > > > >> > >>>>>>>>>>>> tongue-twister,
> > > > >> > >>>>>>>>>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>>>> I
> > > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> > > > >> > >>> pronounce
> > > > >> > >>>>>> it
> > > > >> > >>>>>>>>>>> properly.
> > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>> J.
> > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent
> Bovenzi
> > > > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > > > >> > >>>>>>>> brent@astronomer.io.inva
> > > > >> > >>>>>>>>>> lid>
> > > > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
> > > > >> > >>>>> starting
> > > > >> > >>>>>> on
> > > > >> > >>>>>>>>> AIP-38
> > > > >> > >>>>>>>>>>> to
> > > > >> > >>>>>>>>>>>>>>>>>>> modernize
> > > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are
> > more
> > > > >> > >>>>>>>>> descriptive of
> > > > >> > >>>>>>>>>>>> what
> > > > >> > >>>>>>>>>>>>>>>> one
> > > > >> > >>>>>>>>>>>>>>>>>>> is
> > > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic
> Graph
> > > > >> > >>> is an
> > > > >> > >>>>>>>>>>> implementation
> > > > >> > >>>>>>>>>>>>>>>>> detail.
> > > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
> > > > >> > >>> point
> > > > >> > >>>>>> "DAG"
> > > > >> > >>>>>>>> has
> > > > >> > >>>>>>>>>>> become
> > > > >> > >>>>>>>>>>>>>>>> "dag"
> > > > >> > >>>>>>>>>>>>>>>>>>> , a
> > > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
> > > > >> > >>>>>> discussion,
> > > > >> > >>>>>>>>> Airflow
> > > > >> > >>>>>>>>>>>>>>>> Summit
> > > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI.
> > Let's
> > > > >> > >>>>>> embrace
> > > > >> > >>>>>>>>> "dag".
> > > > >> > >>>>>>>>>>> A
> > > > >> > >>>>>>>>>>>>>>> user
> > > > >> > >>>>>>>>>>>>>>>>>>> just
> > > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the
> technical
> > > > >> > >>>>> concept
> > > > >> > >>>>>>>>> behind
> > > > >> > >>>>>>>>>>> that
> > > > >> > >>>>>>>>>>>>>>>>> word. I
> > > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > > > >> > >>> refactoring so
> > > > >> > >>>>>>> much
> > > > >> > >>>>>>>>> code,
> > > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
> > > > >> > >>>>> questions,
> > > > >> > >>>>>>>> etc.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> > > > >> > >>> Standish
> > > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva
> > <mailto:
> > > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a
> > lot
> > > > >> > >>> of
> > > > >> > >>>>>> pain
> > > > >> > >>>>>>>> for
> > > > >> > >>>>>>>>>>>>>>>>> questionable
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
> > > > >> > >>> idea. I
> > > > >> > >>>>>>> agree
> > > > >> > >>>>>>>>> the
> > > > >> > >>>>>>>>>>>>>>>>>>>>> association
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not
> > helpful.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some
> > less
> > > > >> > >>>>>>> invasive
> > > > >> > >>>>>>>>>>>>>>> mitigations.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
> > > > >> > >>>>>> de-emphasize
> > > > >> > >>>>>>>> the
> > > > >> > >>>>>>>>>>> math
> > > > >> > >>>>>>>>>>>>>>> nerd
> > > > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> > > > >> > >>> website /
> > > > >> > >>>>>> etc,
> > > > >> > >>>>>>>>> *never
> > > > >> > >>>>>>>>>>>>>>>> define*
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed
> > acyclic
> > > > >> > >>>>>> graph.
> > > > >> > >>>>>>>>> Always
> > > > >> > >>>>>>>>>>>>>>> define
> > > > >> > >>>>>>>>>>>>>>>>> it
> > > > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow
> > etc.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it
> is
> > > > >> > >>> like a
> > > > >> > >>>>>>>>> historical
> > > > >> > >>>>>>>>>>>>>>>>> footnote,
> > > > >> > >>>>>>>>>>>>>>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
> > > > >> > >>> hidden.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in
> the
> > > > >> > >>> docs
> > > > >> > >>>>> in
> > > > >> > >>>>>>>>> general
> > > > >> > >>>>>>>>>>> e.g.
> > > > >> > >>>>>>>>>>>>>>>>>>> writing
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> > > > >> > >>> "DAGs"
> > > > >> > >>>>>> etc.
> > > > >> > >>>>>>>> The
> > > > >> > >>>>>>>>>>>> upper
> > > > >> > >>>>>>>>>>>>>>>> case
> > > > >> > >>>>>>>>>>>>>>>>>>>>> part
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
> > > > >> > >>> "dag" in
> > > > >> > >>>>>>>>> airlfow is
> > > > >> > >>>>>>>>>>>> just
> > > > >> > >>>>>>>>>>>>>>>> an
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
> > > > >> > >>> "DAGs"
> > > > >> > >>>>> is
> > > > >> > >>>>>>> not
> > > > >> > >>>>>>>>> really
> > > > >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in
> > airflow
> > > > >> > >>> is
> > > > >> > >>>>> its
> > > > >> > >>>>>>> own
> > > > >> > >>>>>>>>>>> thing,
> > > > >> > >>>>>>>>>>>> is
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
> > > > >> > >>> nobody
> > > > >> > >>>>>> knows
> > > > >> > >>>>>>>>> about
> > > > >> > >>>>>>>>>>>>>>>> anyway),
> > > > >> > >>>>>>>>>>>>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
> > > > >> > >>> normal
> > > > >> > >>>>>>> people
> > > > >> > >>>>>>>>>>>>>>> understand.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek
> > Potiuk
> > > > >> > >>> <
> > > > >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > > > >> > >>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that
> it
> > > > >> > >>> will
> > > > >> > >>>>> be
> > > > >> > >>>>>>>>> extremely
> > > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think
> it
> > > > >> > >>> will
> > > > >> > >>>>>> make
> > > > >> > >>>>>>> a
> > > > >> > >>>>>>>>> lot of
> > > > >> > >>>>>>>>>>>>>>>>>>> "google"
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches
> not
> > > > >> > >>>>> finding
> > > > >> > >>>>>>> the
> > > > >> > >>>>>>>>> right
> > > > >> > >>>>>>>>>>>>>>>>>>> answers.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> This
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> > > > >> > >>> besides the
> > > > >> > >>>>>>>>> community
> > > > >> > >>>>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>>> ideas
> > > > >> > >>>>>>>>>>>>>>>>>>>>> that
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
> > > > >> > >>>>> examples,
> > > > >> > >>>>>>>>> problems
> > > > >> > >>>>>>>>>>> and
> > > > >> > >>>>>>>>>>>>>>>>>>>>> solutions
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> > > > >> > >>> remember
> > > > >> > >>>>>> that
> > > > >> > >>>>>>>>> all the
> > > > >> > >>>>>>>>>>>> AI
> > > > >> > >>>>>>>>>>>>>>>>>>>>> trained
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> on
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> > > > >> > >>> matching
> > > > >> > >>>>>>> queries
> > > > >> > >>>>>>>> of
> > > > >> > >>>>>>>>>>>> people.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could
> > easily
> > > > >> > >>>>>> switch.
> > > > >> > >>>>>>>>> And if
> > > > >> > >>>>>>>>>>> we
> > > > >> > >>>>>>>>>>>>>>> do
> > > > >> > >>>>>>>>>>>>>>>> -
> > > > >> > >>>>>>>>>>>>>>>>>>> I
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
> > > > >> > >>> instead
> > > > >> > >>>>> of
> > > > >> > >>>>>>>>> `dag` if
> > > > >> > >>>>>>>>>>>> not
> > > > >> > >>>>>>>>>>>>>>>> the
> > > > >> > >>>>>>>>>>>>>>>>>>>>> above
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
> > > > >> > >>> that it
> > > > >> > >>>>>>> might
> > > > >> > >>>>>>>>> cause
> > > > >> > >>>>>>>>>>>> more
> > > > >> > >>>>>>>>>>>>>>>>>>>>> problems
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others
> will
> > > > >> > >>> think
> > > > >> > >>>>>>> it's
> > > > >> > >>>>>>>> a
> > > > >> > >>>>>>>>> good
> > > > >> > >>>>>>>>>>>>>>>> idea, I
> > > > >> > >>>>>>>>>>>>>>>>>>>>> am
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> ok
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> > > > >> > >>> Bhakat
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> > > > >> > >>> <mailto:
> > > > >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very
> less
> > > > >> > >>> sense
> > > > >> > >>>>> to
> > > > >> > >>>>>>>>> someone
> > > > >> > >>>>>>>>>>> new
> > > > >> > >>>>>>>>>>>>>>> to
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show
> the
> > > > >> > >>> nature
> > > > >> > >>>>>> of
> > > > >> > >>>>>>>>> being
> > > > >> > >>>>>>>>>>>>>>> acyclic.
> > > > >> > >>>>>>>>>>>>>>>>>>>>> Sure,
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> as
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> > > > >> > >>> workaround it.
> > > > >> > >>>>>>>> Still,
> > > > >> > >>>>>>>>> in
> > > > >> > >>>>>>>>>>> my
> > > > >> > >>>>>>>>>>>>>>>>>>>>> opinion,
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> there
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic
> behavior
> > in
> > > > >> > >>>>>>> workflow
> > > > >> > >>>>>>>>>>>>>>>>>>> orchestration.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> Most
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can
> be
> > > > >> > >>>>> covered
> > > > >> > >>>>>>>>> using an
> > > > >> > >>>>>>>>>>>>>>>> acyclic
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> manner
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> > > > >> > >>> idempotency. So
> > > > >> > >>>>> I
> > > > >> > >>>>>>>> would
> > > > >> > >>>>>>>>> want
> > > > >> > >>>>>>>>>>>> the
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > >> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> > > > >> > >>> bernd.stroehle@kosakya.de>>
> > > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become
> an
> > > > >> > >>>>> Airflow
> > > > >> > >>>>>>> Fan;
> > > > >> > >>>>>>>>> so
> > > > >> > >>>>>>>>>>> many
> > > > >> > >>>>>>>>>>>>>>> new
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should
> be
> > > > >> > >>>>>> replaced
> > > > >> > >>>>>>> by
> > > > >> > >>>>>>>>> the
> > > > >> > >>>>>>>>>>>> more
> > > > >> > >>>>>>>>>>>>>>>>>>>>> general
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or
> > Airflow
> > > > >> > >>>>>> (Petri)
> > > > >> > >>>>>>>> Net
> > > > >> > >>>>>>>>>>> (AFN)
> > > > >> > >>>>>>>>>>>>>>>>>>> (maybe
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored
> > in a
> > > > >> > >>>>> Graph
> > > > >> > >>>>>>>>> Database.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> > > > >> > >>> Graph
> > > > >> > >>>>>> (AFG)
> > > > >> > >>>>>>>>> might be
> > > > >> > >>>>>>>>>>>>>>>>>>>>> assigned
> > > > >> > >>>>>>>>>>>>>>>>>>>>>> to
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>> an
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... )
> member
> > > > >> > >>> of a
> > > > >> > >>>>>>>> library.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
> > > > >> > >>>>> structure
> > > > >> > >>>>>>>> than
> > > > >> > >>>>>>>>> a
> > > > >> > >>>>>>>>>>>>>>>>>>>>> configuration
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's
> bullshit.
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > >> > >>>>>>>>>>> bernd.stroehle@gmail.com>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> > > > >> > >>> <ikholopov@google.com.INVA
> > > > >> > >>>>>>>>> <mailto:
> > > > >> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02
> PM
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > >> > >>>>>>>>> dev@airflow.apache.org
> > > > >> > >>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate
> > the
> > > > >> > >>> term
> > > > >> > >>>>>>> "DAG"
> > > > >> > >>>>>>>>> for
> > > > >> > >>>>>>>>>>> end
> > > > >> > >>>>>>>>>>>>>>>>>>> users
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
> > > > >> > >>>>>> suboptimal,
> > > > >> > >>>>>>> it
> > > > >> > >>>>>>>>> is
> > > > >> > >>>>>>>>>>> part
> > > > >> > >>>>>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Brent Bovenzi <br...@astronomer.io.INVALID>.
-0.5
Even though the new UI prefers "Dag", I don't think that means "DAG" is
incorrect.
On Tue, Feb 18, 2025 at 11:36âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> I love "Dynamic Airflow Graph" +10 to that Constance.
>
> And yes. I fully agree with your "somewhat" acyclic statement. I saw people
> mentioning "event-driven" workflows to better describe the AI/ML, but that
> for me is a synonym of "completely unmanageable workflows" (similar to
> micro-services) - what I was alluding to was exactly this - something like
> curren DAG with parts of it being potentially cyclic - so we are completely
> aligned here :). And that's what make it "not DAG - in mathematical sense"
> - even more reason to depart from it,
>
> J.
>
> On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
> <co...@astronomer.io.invalid> wrote:
>
> > Hey all,
> >
> > I appreciate the discussion about how we position the concept of a DAG in
> > Airflow. While I agree with updating documentation to de-emphasize the
> > strict mathematical definition, I do not think we should change the
> public
> > interface to use "dag" instead of "DAG" everywhere. It adds
> > unnecessary churn and breaks consistency across the ecosystem.
> >
> > Regarding the broader discussion on whether DAGs in Airflow should remain
> > acyclic: I acknowledge that some workflows, particularly in ML and AI,
> > require cyclical behaviour (example: iterative refinement until a
> condition
> > is met). However, I think it's important to recognize that even in those
> > cases, the workflow itself still has an overall start and end. It's not
> an
> > unbounded loop but rather a controlled execution process that happens to
> > have cycles *within* it.
> >
> > That being said, I do see value in reinterpreting "DAG" to better align
> > with how Airflow is actually used, and love shifting away from "Directed
> > Acyclic Graph" in favour of something more intuitive. However "Data
> Airflow
> > Graph" makes it sound like we're defining Airflow itself rather than
> > describing the structure of workflows within it.
> >
> > If we want a new meaning for DAG that resonates with our base while
> keeping
> > Airflow's identity clear, I am proposing three alternatives:
> >
> >
> > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while emphasizing
> > flexibility
> > 2. *Distributed Automation Graph*: Highlights execution across systems
> > and automations without tying it strictly to data
> > 3. *Directed Activity Graph*: Retains the useful meaning of "Directed"
> > while shifting away from strict acyclic constraints. (My personal
> > favourite)
> >
> > I think any shift in terminology should be driven by what best
> communicates
> > Airflow's role workflow automation, rather than just distancing ourselves
> > from the mathematical definition.
> >
> > Curious to hear more thoughts!
> >
> > On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Sorry for the few typos ... My slight dyslexia did not help and
> > autocorrect
> > > did not help either this time : dept -> debt, routed -> rooted and a
> few
> > > other small typos :) .
> > >
> > > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > First of all - yes, agree with Lee and Ash, I think we do not need to
> > > > change the public interface, I'd also be -0 on that, but docs,
> internal
> > > > strings, yes we could.
> > > >
> > > > But .... After thinking a bit and looking at the discussion here,
> where
> > > > people have concerns, I have an alternative take. And I know this
> might
> > > be
> > > > controversial, and some people might find it confusing, but why don't
> > we
> > > > come up with a new acronym for DAG, one that will be more airflow and
> > > less
> > > > "mathematics/algorithm". And I have, I think a good reason for it,
> and
> > I
> > > > even have a proposal. Bear with me,
> > > >
> > > > DAG => *Data Airflow Graph*
> > > >
> > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now
> (with
> > > > exception of some niche products - you know which one I speak about -
> > > which
> > > > are unfortunate enough to use the "Dag" in their name - I will
> explain
> > > why
> > > > this is unfortunate, IMHO).
> > > >
> > > > We are lucky to have Airflow for "A" in the acronym and we can make
> it
> > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow.
> > > >
> > > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > > >
> > > > Not only because "Directed" and "Acyclic" are mostly not in the
> > > vocabulary
> > > > of most of our users - as Ryan, starting this thread explained yes, I
> > > agree
> > > > with it, but also I think there is a more important reason - I
> simply
> > > > believe in the future our DAGs might not (and likely will not) be
> > > "acyclic".
> > > >
> > > > I know for some it might be herezy and we have not discussed any of
> > that
> > > > yet, but more and more I keep on hearing that Machine Learning
> > Workflows
> > > of
> > > > the future are often somewhat cyclic. In our case there is one type
> of
> > > > repeat-in-cycle behaviour - we have "task retry on failure" - but
> this
> > is
> > > > only on failure and it basically mens "idempotent" retry - we want to
> > > redo
> > > > something and get single result out of that and basically discard the
> > > > previous result. But machine learning/AI workflows are different. I
> > had a
> > > > lot of conversations recently (I even co-organized "AI low-level
> > > > engineering and hacking" devroom to learn more about those) - and I
> > keep
> > > on
> > > > hearing that "Repeat a sequence of things several times and refine"
> is
> > > > precisely what AI / ML workflows will need - especially with Agentic
> > > > workflows - often we have cooperating agents which **could** be part
> of
> > > the
> > > > same DAG, but currently in Airflow where our DAGs are "Acyclic" we
> can
> > > > only express "repeat several time and refine" as a single task that
> > will
> > > do
> > > > such refining. If we would like to split such an "agent" doing their
> > own
> > > > work in "cycles" and communicating with other "agents" doing the
> same -
> > > > represented as a multi-task-entity, we have no way to express it
> using
> > > our
> > > > current "acyclic" semantics.
> > > >
> > > > But - other than our implementation makes that assumption currently -
> > > > there is nothing fundamental with Airflow being able to run such
> > > "partially
> > > > cyclic workflows". We have all the pieces, we would have to change
> > > > dependency calculation to allow for cycles, update UI to handle
> cycles
> > > > representation - and we could in theory quite seamlessly, support
> > > "cyclic"
> > > > workflows.
> > > >
> > > > I know, I know, that's the first time we hear it and that's huge
> > change -
> > > > of course, I do not want to start discussing any details of it, not
> > > before
> > > > Airflow 3.0 is out - but I think we should be open that at some point
> > of
> > > > time the "Acyclic" property is going to go away, And my personal
> > > intuition
> > > > tells me that a year from now we will have acyclic workflows
> (assuming
> > > that
> > > > all the foundational work we do on Airflow 3 will really help in
> > > decreasing
> > > > all the technical dept, and will open our minds to new ideas).
> > > >
> > > > That's why I think - the sooner we get rid of "Directed Acyclic" -
> the
> > > > better. that will - even for us, maintainer free us from being routed
> > in
> > > > the acyclic nature of Airflow.
> > > >
> > > > How about we change to "Data Airflow Graph" (or any other good
> acronym
> > > > which has (A)irflow instead of (A)cyclic). The name is just a
> proposal.
> > > >
> > > > Now.. I go hide somewhere as I believe that might cause a "flamewar"
> > :),
> > > > unless on request of Ryan you already sheathed your swords in the
> > > > discussion (which seems everyone did).
> > > >
> > > > But honestly - I would really love what you think of "let's just
> break
> > > > with Acyclic NOW" and shape our future by using a different acronym.
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > > ephraimanierobi@gmail.com> wrote:
> > > >
> > > >> When I started contributing to Airflow, I had to read up on DAG, and
> > > now I
> > > >> know what it means. I wonder if we are about to have users who use
> the
> > > >> term
> > > >> Dag without knowing that there's a DAG.
> > > >>
> > > >> Slightly concerned.
> > > >>
> > > >> -ephraim
> > > >>
> > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com> wrote:
> > > >>
> > > >> > For user-facing things (e.g., `from airflow import DAG`), Iâm more
> > > like
> > > >> > -0. But for documentation, docstring, internal things and etc., we
> > > >> probably
> > > >> > could still change most of them?
> > > >> >
> > > >> > Best,
> > > >> > Wei
> > > >> >
> > > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org>
> > > >> wrote:
> > > >> > >
> > > >> > > I also donât personally think itâs worth the pain (not to
> mention
> > > >> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on
> that.
> > > >> > >
> > > >> > > -ash
> > > >> > >
> > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > > daniel.imberman@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> > >>
> > > >> > >> I think my biggest concern is a marketing one and not a
> technical
> > > >> one.
> > > >> > >>
> > > >> > >> As has been mentioned on the thread the terms airflow and dag
> are
> > > >> kind
> > > >> > of
> > > >> > >> synonymous and I certainly donât want to give the impression
> that
> > > we
> > > >> are
> > > >> > >> breaking more than we are breaking.
> > > >> > >>
> > > >> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> > > >> > >>
> > > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org>
> > wrote:
> > > >> > >>
> > > >> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff
> rules
> > > are
> > > >> > best
> > > >> > >>> suited for user-facing things but not the airflow code base
> > > itself.
> > > >> If
> > > >> > what
> > > >> > >>> we mean is adding a rule to avoid users using "DAG" *after* we
> > > >> rename
> > > >> > it,
> > > >> > >>> it's definitely a +1000.
> > > >> > >>>
> > > >> > >>> I just created GitHub issues for this removing "DAG"
> > > >> > >>>
> > > >> > >>> https://github.com/apache/airflow/issues/46842
> > > >> > >>> https://github.com/apache/airflow/issues/46843
> > > >> > >>>
> > > >> > >>> One thing I'm not sure about is whether we want to get rid of
> > > `from
> > > >> > >>> airflow import DAG` as well. đ€
> > > >> > >>>
> > > >> > >>> Best,
> > > >> > >>> Wei
> > > >> > >>>
> > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > > >> > >>>> Hard to say until it's looked at :)
> > > >> > >>>>
> > > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > > >> aritrabasu1999@gmail.com
> > > >> > >
> > > >> > >>>> wrote:
> > > >> > >>>>
> > > >> > >>>>> I can take it up, it's mostly just a doc update right? Or
> are
> > we
> > > >> > doing
> > > >> > >>> code
> > > >> > >>>>> files replacement too?
> > > >> > >>>>> --
> > > >> > >>>>> Regards,
> > > >> > >>>>> Aritra Basu
> > > >> > >>>>>
> > > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <
> jarek@potiuk.com
> > >
> > > >> > wrote:
> > > >> > >>>>>
> > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any
> > > >> volunteers
> > > >> > >>> to
> > > >> > >>>>>> make it ? It's fun.
> > > >> > >>>>>>
> > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > > droiddev510e@gmail.com>
> > > >> > >>> wrote:
> > > >> > >>>>>>
> > > >> > >>>>>>> +1 for ruff rules :)
> > > >> > >>>>>>>
> > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in
> > the
> > > >> > >>> Airflow
> > > >> > >>>>>>> docs, in line with the new UI changes and to make the
> > renaming
> > > >> > >>>>>>> consistent across user-facing pages.
> > > >> > >>>>>>>
> > > >> > >>>>>>> Regards,
> > > >> > >>>>>>> Omkar
> > > >> > >>>>>>>
> > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > > jarek@potiuk.com>
> > > >> > >>> wrote:
> > > >> > >>>>>>>
> > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> > > >> > >>>>>>>>
> > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <
> weilee@apache.org
> > >
> > > >> > >>> wrote:
> > > >> > >>>>>>>>
> > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag" or
> > > "Dag"
> > > >> > >>>>> instead
> > > >> > >>>>>>> of
> > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in
> > the
> > > >> > >>>>> codebase
> > > >> > >>>>>>> with
> > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it
> might
> > be
> > > >> > >>> đ€),
> > > >> > >>>>>>> should
> > > >> > >>>>>>>> we
> > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the following
> PRs?
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>> Best,
> > > >> > >>>>>>>>> Wei
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > > >> > >>>>>>>>>> Let me know if that's the direction we want to go.
> > > >> > >>>>>>>>>>
> > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > > >> > >>>>>>>>>>
> > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
> > > >> > >>> synonymous
> > > >> > >>>>>> with
> > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and
> explain
> > > >> > >>> what
> > > >> > >>>>>>> Airflow
> > > >> > >>>>>>>>> is
> > > >> > >>>>>>>>>>> all about. Although folks would ask, I donât ever use
> > the
> > > >> > >>>>>>>> mathematical
> > > >> > >>>>>>>>>>> definition of DAG.
> > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for many
> > > >> > >>> users
> > > >> > >>>>>> making
> > > >> > >>>>>>>>> such a
> > > >> > >>>>>>>>>>> change - I would rather direct that energy to
> appending
> > > the
> > > >> > >>>>>> Oxford
> > > >> > >>>>>>>>>>> definition of "dag" to include a reference to
> workflows
> > in
> > > >> > >>>>>> Airflow.
> > > >> > >>>>>>>> __
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> -- Rajesh
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> > jarek@potiuk.com
> > > >> > >>>>>> <mailto:
> > > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the
> > > >> > >>>>> organization.
> > > >> > >>>>>> Do
> > > >> > >>>>>>>> not
> > > >> > >>>>>>>>>>> click links or open attachments unless you can confirm
> > the
> > > >> > >>>>> sender
> > > >> > >>>>>>> and
> > > >> > >>>>>>>>> know
> > > >> > >>>>>>>>>>> the content is safe.
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > > >> > >>>>> expéditeur
> > > >> > >>>>>>>>> externe.
> > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > jointe
> > > >> > >>> si
> > > >> > >>>>> vous
> > > >> > >>>>>>> ne
> > > >> > >>>>>>>>> pouvez
> > > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous
> > nâĂȘtes
> > > >> > >>> pas
> > > >> > >>>>>>>> certain
> > > >> > >>>>>>>>> que
> > > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> From Guido's post:
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been
> > > renamed".
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> Naming is hard.
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> J.
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > >> > >>>>> droiddev510e@gmail.com
> > > >> > >>>>>>>>> <mailto:
> > > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an
> > > >> > >>> Airflow
> > > >> > >>>>>> term
> > > >> > >>>>>>>> now
> > > >> > >>>>>>>>>>>> rather
> > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or
> > not,
> > > >> > >>> I've
> > > >> > >>>>>>>> worked
> > > >> > >>>>>>>>> with
> > > >> > >>>>>>>>>>>> work
> > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now
> > > >> > >>> and are
> > > >> > >>>>>> pro
> > > >> > >>>>>>>>> devs
> > > >> > >>>>>>>>>>> but
> > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For
> > > >> > >>> them,
> > > >> > >>>>> dag
> > > >> > >>>>>> =
> > > >> > >>>>>>>>> Airflow.
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
> > > >> > >>> will be
> > > >> > >>>>> a
> > > >> > >>>>>>>> major
> > > >> > >>>>>>>>>>>> behavior
> > > >> > >>>>>>>>>>>> change for the developer community, so we'll need a
> > solid
> > > >> > >>>>> plan
> > > >> > >>>>>> on
> > > >> > >>>>>>>>> how to
> > > >> > >>>>>>>>>>>> introduce it, when we do.
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the
> > Great
> > > >> > >>>>>> Renaming
> > > >> > >>>>>>>> in
> > > >> > >>>>>>>>> core
> > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could
> probably
> > > >> > >>> get
> > > >> > >>>>>> some
> > > >> > >>>>>>>>>>> learnings
> > > >> > >>>>>>>>>>>> from there? Who knows!
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>> Guido's blog (2009):
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>
> > > >> > >>>>>
> > > >> > >>>
> > > >> >
> > > >>
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > >> > >>>>>>>>>>> <
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>
> > > >> > >>>>>
> > > >> > >>>
> > > >> >
> > > >>
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>
> > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > >> > >>>>>>> <
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>
> > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > >
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>> Cheers,
> > > >> > >>>>>>>>>>>> Omkar
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > > >> > >>> weilee.rx@gmail.com
> > > >> > >>>>>>>>> <mailto:
> > > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an
> > airflow
> > > >> > >>>>> term.
> > > >> > >>>>>>> At
> > > >> > >>>>>>>>> least,
> > > >> > >>>>>>>>>>>>> thatâs how I understood it when I first used
> Airflow.
> > I
> > > >> > >>>>> feel
> > > >> > >>>>>>>>> renaming
> > > >> > >>>>>>>>>>> it
> > > >> > >>>>>>>>>>>> at
> > > >> > >>>>>>>>>>>>> this stage would require considerable effort from
> > > >> > >>>>> maintainers
> > > >> > >>>>>>> and
> > > >> > >>>>>>>>>>>> existing
> > > >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > > >> > >>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>> Best,
> > > >> > >>>>>>>>>>>>> Wei
> > > >> > >>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > >> > >>>>>> kaxilnaik@gmail.com
> > > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > > >> > >>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we
> re-kindle
> > > >> > >>>>> this
> > > >> > >>>>>>>>> discussion
> > > >> > >>>>>>>>>>>> for
> > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too
> much
> > > >> > >>>>>>> disruption
> > > >> > >>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > > >> > >>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
> > > >> > >>>>>> experience
> > > >> > >>>>>>>>> what a
> > > >> > >>>>>>>>>>>> dag
> > > >> > >>>>>>>>>>>>> is,
> > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes -
> > > >> > >>> stuff
> > > >> > >>>>>> like
> > > >> > >>>>>>>>> dags
> > > >> > >>>>>>>>>>>> being
> > > >> > >>>>>>>>>>>>> a
> > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The
> structure
> > > >> > >>>>>> doesn't
> > > >> > >>>>>>>>> come up.
> > > >> > >>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> > > >> > >>> embrace
> > > >> > >>>>> the
> > > >> > >>>>>>> name
> > > >> > >>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
> > > >> > >>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > > >> > >>>>> vikram@astronomer.io.inva
> > > >> > >>>>>>>>> lid>
> > > >> > >>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> > > >> > >>>>> struggling
> > > >> > >>>>>>> with
> > > >> > >>>>>>>>> this
> > > >> > >>>>>>>>>>>>> when I
> > > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being
> an
> > > >> > >>>>>>>> established
> > > >> > >>>>>>>>>>>> concept
> > > >> > >>>>>>>>>>>>>>> now
> > > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > > >> > >>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
> > > >> > >>> best
> > > >> > >>>>>>>> expressed
> > > >> > >>>>>>>>> by
> > > >> > >>>>>>>>>>>>> Daniel
> > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
> > > >> > >>> than
> > > >> > >>>>>> the
> > > >> > >>>>>>>>> computer
> > > >> > >>>>>>>>>>>>>>> science
> > > >> > >>>>>>>>>>>>>>>> concept.
> > > >> > >>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>> Best regards,
> > > >> > >>>>>>>>>>>>>>>> Vikram
> > > >> > >>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > > >> > >>>>>> onikolas@amazon.com.inva
> > > >> > >>>>>>>>> lid
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
> > > >> > >>> right
> > > >> > >>>>>> Ryan,
> > > >> > >>>>>>>> DAG
> > > >> > >>>>>>>>>>> isn't
> > > >> > >>>>>>>>>>>>>>> great
> > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
> > > >> > >>> cause
> > > >> > >>>>> much
> > > >> > >>>>>>>> more
> > > >> > >>>>>>>>>>>> wreckage
> > > >> > >>>>>>>>>>>>>>>> than
> > > >> > >>>>>>>>>>>>>>>>> it solves.
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
> > > >> > >>>>> defining
> > > >> > >>>>>>>> DAG.
> > > >> > >>>>>>>>> I
> > > >> > >>>>>>>>>>>> think
> > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
> > > >> > >>> for a
> > > >> > >>>>>> while
> > > >> > >>>>>>>> now
> > > >> > >>>>>>>>>>>> anyway,
> > > >> > >>>>>>>>>>>>>>> so
> > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> Cheers,
> > > >> > >>>>>>>>>>>>>>>>> Niko
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> ________________________________
> > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org
> <mailto:
> > > >> > >>>>>>>>> ash@apache.org>>
> > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > >> > >>>>>> dev@airflow.apache.org
> > > >> > >>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
> > > >> > >>> term
> > > >> > >>>>>> "DAG"
> > > >> > >>>>>>>>> for end
> > > >> > >>>>>>>>>>>>>>> users
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of
> the
> > > >> > >>>>>>>>> organization.
> > > >> > >>>>>>>>>>> Do
> > > >> > >>>>>>>>>>>>> not
> > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can
> > > >> > >>> confirm
> > > >> > >>>>>> the
> > > >> > >>>>>>>>> sender
> > > >> > >>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>> know
> > > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> > > >> > >>> dâun
> > > >> > >>>>>>>>> expéditeur
> > > >> > >>>>>>>>>>>>>>> externe.
> > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune
> piĂšce
> > > >> > >>>>>> jointe
> > > >> > >>>>>>> si
> > > >> > >>>>>>>>> vous
> > > >> > >>>>>>>>>>> ne
> > > >> > >>>>>>>>>>>>>>>> pouvez
> > > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> > > >> > >>> vous
> > > >> > >>>>>> nâĂȘtes
> > > >> > >>>>>>>> pas
> > > >> > >>>>>>>>>>>> certain
> > > >> > >>>>>>>>>>>>>>>> que
> > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
> > > >> > >>> term â
> > > >> > >>>>>>>> getting
> > > >> > >>>>>>>>> to
> > > >> > >>>>>>>>>>>>> re-use
> > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way,
> both
> > > >> > >>>>> sides
> > > >> > >>>>>>> have
> > > >> > >>>>>>>>> good
> > > >> > >>>>>>>>>>>>>>> reasons
> > > >> > >>>>>>>>>>>>>>>>> presented.
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>> -a
> > > >> > >>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
> > > >> > >>> from,
> > > >> > >>>>>>> "well,
> > > >> > >>>>>>>>> no
> > > >> > >>>>>>>>>>> one
> > > >> > >>>>>>>>>>>>>>>>> actually
> > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> > > >> > >>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > > >> > >>>>>>>>> jarek@potiuk.com
> > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > >> > >>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
> > > >> > >>> the
> > > >> > >>>>>>>>>>> tongue-twister,
> > > >> > >>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > > >> > >>>>>>>>> jarek@potiuk.com
> > > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > > >> > >>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
> > > >> > >>> would
> > > >> > >>>>> very
> > > >> > >>>>>>>> much
> > > >> > >>>>>>>>> want
> > > >> > >>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>> be
> > > >> > >>>>>>>>>>>>>>>>>>> able
> > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> > > >> > >>> further.
> > > >> > >>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
> > > >> > >>> it's a
> > > >> > >>>>>>>>>>>> tongue-twister,
> > > >> > >>>>>>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>> I
> > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> > > >> > >>> pronounce
> > > >> > >>>>>> it
> > > >> > >>>>>>>>>>> properly.
> > > >> > >>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>> J.
> > > >> > >>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > > >> > >>>>>>>> brent@astronomer.io.inva
> > > >> > >>>>>>>>>> lid>
> > > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
> > > >> > >>>>> starting
> > > >> > >>>>>> on
> > > >> > >>>>>>>>> AIP-38
> > > >> > >>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>> modernize
> > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are
> more
> > > >> > >>>>>>>>> descriptive of
> > > >> > >>>>>>>>>>>> what
> > > >> > >>>>>>>>>>>>>>>> one
> > > >> > >>>>>>>>>>>>>>>>>>> is
> > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
> > > >> > >>> is an
> > > >> > >>>>>>>>>>> implementation
> > > >> > >>>>>>>>>>>>>>>>> detail.
> > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
> > > >> > >>> point
> > > >> > >>>>>> "DAG"
> > > >> > >>>>>>>> has
> > > >> > >>>>>>>>>>> become
> > > >> > >>>>>>>>>>>>>>>> "dag"
> > > >> > >>>>>>>>>>>>>>>>>>> , a
> > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
> > > >> > >>>>>> discussion,
> > > >> > >>>>>>>>> Airflow
> > > >> > >>>>>>>>>>>>>>>> Summit
> > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI.
> Let's
> > > >> > >>>>>> embrace
> > > >> > >>>>>>>>> "dag".
> > > >> > >>>>>>>>>>> A
> > > >> > >>>>>>>>>>>>>>> user
> > > >> > >>>>>>>>>>>>>>>>>>> just
> > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
> > > >> > >>>>> concept
> > > >> > >>>>>>>>> behind
> > > >> > >>>>>>>>>>> that
> > > >> > >>>>>>>>>>>>>>>>> word. I
> > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > > >> > >>> refactoring so
> > > >> > >>>>>>> much
> > > >> > >>>>>>>>> code,
> > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
> > > >> > >>>>> questions,
> > > >> > >>>>>>>> etc.
> > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> > > >> > >>> Standish
> > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva
> <mailto:
> > > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a
> lot
> > > >> > >>> of
> > > >> > >>>>>> pain
> > > >> > >>>>>>>> for
> > > >> > >>>>>>>>>>>>>>>>> questionable
> > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
> > > >> > >>> idea. I
> > > >> > >>>>>>> agree
> > > >> > >>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>> association
> > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not
> helpful.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some
> less
> > > >> > >>>>>>> invasive
> > > >> > >>>>>>>>>>>>>>> mitigations.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
> > > >> > >>>>>> de-emphasize
> > > >> > >>>>>>>> the
> > > >> > >>>>>>>>>>> math
> > > >> > >>>>>>>>>>>>>>> nerd
> > > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> > > >> > >>> website /
> > > >> > >>>>>> etc,
> > > >> > >>>>>>>>> *never
> > > >> > >>>>>>>>>>>>>>>> define*
> > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed
> acyclic
> > > >> > >>>>>> graph.
> > > >> > >>>>>>>>> Always
> > > >> > >>>>>>>>>>>>>>> define
> > > >> > >>>>>>>>>>>>>>>>> it
> > > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow
> etc.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
> > > >> > >>> like a
> > > >> > >>>>>>>>> historical
> > > >> > >>>>>>>>>>>>>>>>> footnote,
> > > >> > >>>>>>>>>>>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
> > > >> > >>> hidden.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
> > > >> > >>> docs
> > > >> > >>>>> in
> > > >> > >>>>>>>>> general
> > > >> > >>>>>>>>>>> e.g.
> > > >> > >>>>>>>>>>>>>>>>>>> writing
> > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> > > >> > >>> "DAGs"
> > > >> > >>>>>> etc.
> > > >> > >>>>>>>> The
> > > >> > >>>>>>>>>>>> upper
> > > >> > >>>>>>>>>>>>>>>> case
> > > >> > >>>>>>>>>>>>>>>>>>>>> part
> > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
> > > >> > >>> "dag" in
> > > >> > >>>>>>>>> airlfow is
> > > >> > >>>>>>>>>>>> just
> > > >> > >>>>>>>>>>>>>>>> an
> > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
> > > >> > >>> "DAGs"
> > > >> > >>>>> is
> > > >> > >>>>>>> not
> > > >> > >>>>>>>>> really
> > > >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in
> airflow
> > > >> > >>> is
> > > >> > >>>>> its
> > > >> > >>>>>>> own
> > > >> > >>>>>>>>>>> thing,
> > > >> > >>>>>>>>>>>> is
> > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
> > > >> > >>> nobody
> > > >> > >>>>>> knows
> > > >> > >>>>>>>>> about
> > > >> > >>>>>>>>>>>>>>>> anyway),
> > > >> > >>>>>>>>>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
> > > >> > >>> normal
> > > >> > >>>>>>> people
> > > >> > >>>>>>>>>>>>>>> understand.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek
> Potiuk
> > > >> > >>> <
> > > >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > > >> > >>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
> > > >> > >>> will
> > > >> > >>>>> be
> > > >> > >>>>>>>>> extremely
> > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
> > > >> > >>> will
> > > >> > >>>>>> make
> > > >> > >>>>>>> a
> > > >> > >>>>>>>>> lot of
> > > >> > >>>>>>>>>>>>>>>>>>> "google"
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
> > > >> > >>>>> finding
> > > >> > >>>>>>> the
> > > >> > >>>>>>>>> right
> > > >> > >>>>>>>>>>>>>>>>>>> answers.
> > > >> > >>>>>>>>>>>>>>>>>>>>>> This
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> > > >> > >>> besides the
> > > >> > >>>>>>>>> community
> > > >> > >>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>> ideas
> > > >> > >>>>>>>>>>>>>>>>>>>>> that
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
> > > >> > >>>>> examples,
> > > >> > >>>>>>>>> problems
> > > >> > >>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>>>> solutions
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> > > >> > >>> remember
> > > >> > >>>>>> that
> > > >> > >>>>>>>>> all the
> > > >> > >>>>>>>>>>>> AI
> > > >> > >>>>>>>>>>>>>>>>>>>>> trained
> > > >> > >>>>>>>>>>>>>>>>>>>>>> on
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> > > >> > >>> matching
> > > >> > >>>>>>> queries
> > > >> > >>>>>>>> of
> > > >> > >>>>>>>>>>>> people.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could
> easily
> > > >> > >>>>>> switch.
> > > >> > >>>>>>>>> And if
> > > >> > >>>>>>>>>>> we
> > > >> > >>>>>>>>>>>>>>> do
> > > >> > >>>>>>>>>>>>>>>> -
> > > >> > >>>>>>>>>>>>>>>>>>> I
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
> > > >> > >>> instead
> > > >> > >>>>> of
> > > >> > >>>>>>>>> `dag` if
> > > >> > >>>>>>>>>>>> not
> > > >> > >>>>>>>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>> above
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
> > > >> > >>> that it
> > > >> > >>>>>>> might
> > > >> > >>>>>>>>> cause
> > > >> > >>>>>>>>>>>> more
> > > >> > >>>>>>>>>>>>>>>>>>>>> problems
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
> > > >> > >>> think
> > > >> > >>>>>>> it's
> > > >> > >>>>>>>> a
> > > >> > >>>>>>>>> good
> > > >> > >>>>>>>>>>>>>>>> idea, I
> > > >> > >>>>>>>>>>>>>>>>>>>>> am
> > > >> > >>>>>>>>>>>>>>>>>>>>>> ok
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> > > >> > >>> Bhakat
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> > > >> > >>> <mailto:
> > > >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
> > > >> > >>> sense
> > > >> > >>>>> to
> > > >> > >>>>>>>>> someone
> > > >> > >>>>>>>>>>> new
> > > >> > >>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
> > > >> > >>> nature
> > > >> > >>>>>> of
> > > >> > >>>>>>>>> being
> > > >> > >>>>>>>>>>>>>>> acyclic.
> > > >> > >>>>>>>>>>>>>>>>>>>>> Sure,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> as
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> > > >> > >>> workaround it.
> > > >> > >>>>>>>> Still,
> > > >> > >>>>>>>>> in
> > > >> > >>>>>>>>>>> my
> > > >> > >>>>>>>>>>>>>>>>>>>>> opinion,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> there
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior
> in
> > > >> > >>>>>>> workflow
> > > >> > >>>>>>>>>>>>>>>>>>> orchestration.
> > > >> > >>>>>>>>>>>>>>>>>>>>>> Most
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
> > > >> > >>>>> covered
> > > >> > >>>>>>>>> using an
> > > >> > >>>>>>>>>>>>>>>> acyclic
> > > >> > >>>>>>>>>>>>>>>>>>>>>> manner
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> > > >> > >>> idempotency. So
> > > >> > >>>>> I
> > > >> > >>>>>>>> would
> > > >> > >>>>>>>>> want
> > > >> > >>>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > >> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> > > >> > >>> bernd.stroehle@kosakya.de>>
> > > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
> > > >> > >>>>> Airflow
> > > >> > >>>>>>> Fan;
> > > >> > >>>>>>>>> so
> > > >> > >>>>>>>>>>> many
> > > >> > >>>>>>>>>>>>>>> new
> > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
> > > >> > >>>>>> replaced
> > > >> > >>>>>>> by
> > > >> > >>>>>>>>> the
> > > >> > >>>>>>>>>>>> more
> > > >> > >>>>>>>>>>>>>>>>>>>>> general
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or
> Airflow
> > > >> > >>>>>> (Petri)
> > > >> > >>>>>>>> Net
> > > >> > >>>>>>>>>>> (AFN)
> > > >> > >>>>>>>>>>>>>>>>>>> (maybe
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored
> in a
> > > >> > >>>>> Graph
> > > >> > >>>>>>>>> Database.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> > > >> > >>> Graph
> > > >> > >>>>>> (AFG)
> > > >> > >>>>>>>>> might be
> > > >> > >>>>>>>>>>>>>>>>>>>>> assigned
> > > >> > >>>>>>>>>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> an
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
> > > >> > >>> of a
> > > >> > >>>>>>>> library.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
> > > >> > >>>>> structure
> > > >> > >>>>>>>> than
> > > >> > >>>>>>>>> a
> > > >> > >>>>>>>>>>>>>>>>>>>>> configuration
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > >> > >>>>>>>>>>> bernd.stroehle@gmail.com>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> > > >> > >>> <ikholopov@google.com.INVA
> > > >> > >>>>>>>>> <mailto:
> > > >> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > >> > >>>>>>>>> dev@airflow.apache.org
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate
> the
> > > >> > >>> term
> > > >> > >>>>>>> "DAG"
> > > >> > >>>>>>>>> for
> > > >> > >>>>>>>>>>> end
> > > >> > >>>>>>>>>>>>>>>>>>> users
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
> > > >> > >>>>>> suboptimal,
> > > >> > >>>>>>> it
> > > >> > >>>>>>>>> is
> > > >> > >>>>>>>>>>> part
> > > >> > >>>>>>>>>>>>>>> of
> > > >> > >>>>>>>>>>>>>>>>>>>>>> Airflow
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
> > > >> > >>> levels,
> > > >> > >>>>> that
> > > >> > >>>>>>> any
> > > >> > >>>>>>>>>>> attempt
> > > >> > >>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>> change
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> it
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
> > > >> > >>> reduce it.
> > > >> > >>>>>> The
> > > >> > >>>>>>>>> only
> > > >> > >>>>>>>>>>>> thing
> > > >> > >>>>>>>>>>>>>>>>>>>>> that is
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> worse
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
> > > >> > >>> when
> > > >> > >>>>>>> there
> > > >> > >>>>>>>>> are
> > > >> > >>>>>>>>>>> two
> > > >> > >>>>>>>>>>>>>>> ways
> > > >> > >>>>>>>>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> define
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
> > > >> > >>>>> tutorials
> > > >> > >>>>>>>> will
> > > >> > >>>>>>>>>>>> suddenly
> > > >> > >>>>>>>>>>>>>>>>>>>>> become
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
> > > >> > >>> as
> > > >> > >>>>>>> "DAG"s.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
> > > >> > >>> users
> > > >> > >>>>> away
> > > >> > >>>>>>>> with
> > > >> > >>>>>>>>> a
> > > >> > >>>>>>>>>>>> number
> > > >> > >>>>>>>>>>>>>>>>>>> of
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> breaking
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even
> more
> > > >> > >>>>>> breaking
> > > >> > >>>>>>>>> changes
> > > >> > >>>>>>>>>>>> for
> > > >> > >>>>>>>>>>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>>> most
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that
> people
> > > >> > >>> are
> > > >> > >>>>>>>> looking
> > > >> > >>>>>>>>> for.
> > > >> > >>>>>>>>>>>>>>>>>>>>> Attempting
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
> > > >> > >>>>> interpreted
> > > >> > >>>>>>> as
> > > >> > >>>>>>>> an
> > > >> > >>>>>>>>>>> even
> > > >> > >>>>>>>>>>>>>>>>>>>>> stronger
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> signal
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of
> changing
> > > >> > >>> the
> > > >> > >>>>>> term
> > > >> > >>>>>>>> in
> > > >> > >>>>>>>>> the
> > > >> > >>>>>>>>>>>> long
> > > >> > >>>>>>>>>>>>>>>>>>>>> run. I
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> even
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
> > > >> > >>> it in
> > > >> > >>>>>> the
> > > >> > >>>>>>>> DAG
> > > >> > >>>>>>>>>>>>>>> definition
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> interface.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
> > > >> > >>> other
> > > >> > >>>>>> places
> > > >> > >>>>>>>> in
> > > >> > >>>>>>>>>>>> Airflow,
> > > >> > >>>>>>>>>>>>>>>>>>> of
> > > >> > >>>>>>>>>>>>>>>>>>>>>> which
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
> > > >> > >>> Harenslak
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > > >> > >>>>>>>> bas@astronomer.io.inva
> > > >> > >>>>>>>>>> lid
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
> > > >> > >>> have
> > > >> > >>>>>>> already
> > > >> > >>>>>>>>> faded
> > > >> > >>>>>>>>>>>> over
> > > >> > >>>>>>>>>>>>>>>>>>>>> time,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways
> to
> > > >> > >>>>>> create
> > > >> > >>>>>>>>> cyclic
> > > >> > >>>>>>>>>>>>>>>>>>> graphs,
> > > >> > >>>>>>>>>>>>>>>>>>>>>> e.g.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I
> imagine
> > > >> > >>>>> these
> > > >> > >>>>>>>>>>> properties
> > > >> > >>>>>>>>>>>>>>>>>>>>>> vanishing
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
> > > >> > >>>>>>> perspective I
> > > >> > >>>>>>>>>>> support
> > > >> > >>>>>>>>>>>>>>>>>>>>>> changing
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks
> do
> > > >> > >>>>>> naming:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
> > > >> > >>> name.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
> > > >> > >>> change, I
> > > >> > >>>>>>> suggest
> > > >> > >>>>>>>>>>>> defining
> > > >> > >>>>>>>>>>>>>>> a
> > > >> > >>>>>>>>>>>>>>>>>>>>>> clear
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
> > > >> > >>> the
> > > >> > >>>>>>>>> deprecation in
> > > >> > >>>>>>>>>>>>>>>>>>>>> Airflow
> > > >> > >>>>>>>>>>>>>>>>>>>>>> 3,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > > >> > >>>>>>>> neil4real@gmail.com
> > > >> > >>>>>>>>>>> <ma...@gmail.com>>
> > > >> > >>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term
> DAG,
> > > >> > >>>>>>>> especially
> > > >> > >>>>>>>>> when
> > > >> > >>>>>>>>>>>>>>>>>>> most
> > > >> > >>>>>>>>>>>>>>>>>>>>>> other
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
> > > >> > >>> wholeheartedly.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
> > > >> > >>>>> confusing
> > > >> > >>>>>>>>> about it
> > > >> > >>>>>>>>>>> at
> > > >> > >>>>>>>>>>>>>>>>>>>>> all,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be
> fairly
> > > >> > >>>>>>> confusing
> > > >> > >>>>>>>>> to
> > > >> > >>>>>>>>>>> most
> > > >> > >>>>>>>>>>>>>>>>>>>>> users
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> who
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM
> Tzu-ping
> > > >> > >>>>> Chung
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > > >> > >>>>>>>>> tp@astronomer.io.inva
> > > >> > >>>>>>>>>>>> lid
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
> > > >> > >>> the
> > > >> > >>>>> term
> > > >> > >>>>>>>> DAG.
> > > >> > >>>>>>>>> The
> > > >> > >>>>>>>>>>>> only
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> problem
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling
> peopleâincluding
> > > >> > >>>>>>> myselfâto
> > > >> > >>>>>>>>> stop
> > > >> > >>>>>>>>>>>> using
> > > >> > >>>>>>>>>>>>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable
> alternative.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
> > > >> > >>> mentioned
> > > >> > >>>>>>>>> âworkflowâ is
> > > >> > >>>>>>>>>>>> not
> > > >> > >>>>>>>>>>>>>>>>>>>>> very
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ
> was
> > > >> > >>>>>>> proposed,
> > > >> > >>>>>>>>> but
> > > >> > >>>>>>>>>>>> itâs a
> > > >> > >>>>>>>>>>>>>>>>>>>>> bit
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
> > > >> > >>> places
> > > >> > >>>>> and
> > > >> > >>>>>>> kept
> > > >> > >>>>>>>>>>> finding
> > > >> > >>>>>>>>>>>>>>>>>>>>> myself
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
> > > >> > >>> wanting to
> > > >> > >>>>>>>> clarify
> > > >> > >>>>>>>>> âDAG
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> definitionâ
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
> > > >> > >>> Scheffler
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > >> > >>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
> > > >> > >>> exactly
> > > >> > >>>>> same
> > > >> > >>>>>>>>>>>> observation,
> > > >> > >>>>>>>>>>>>>>>>>>>>> had a
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
> > > >> > >>> always
> > > >> > >>>>> an
> > > >> > >>>>>>>>>>> introduction
> > > >> > >>>>>>>>>>>>>>>>>>> if
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
> > > >> > >>> makes
> > > >> > >>>>>>> sense.
> > > >> > >>>>>>>>>>>>>>>>>>> Especially
> > > >> > >>>>>>>>>>>>>>>>>>>>>> when
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> also
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also
> a
> > > >> > >>>>>>> reasonable
> > > >> > >>>>>>>>> step.
> > > >> > >>>>>>>>>>>>>>>>>>>>> Concepts
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> can
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join
> hiding
> > > >> > >>>>> below
> > > >> > >>>>>>> the
> > > >> > >>>>>>>>> desk
> > > >> > >>>>>>>>>>>> with
> > > >> > >>>>>>>>>>>>>>>>>>>>> you
> > > >> > >>>>>>>>>>>>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
> > > >> > >>> keep
> > > >> > >>>>>>>> details
> > > >> > >>>>>>>>> of
> > > >> > >>>>>>>>>>>>>>>>>>> python
> > > >> > >>>>>>>>>>>>>>>>>>>>>> names
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
> > > >> > >>> DAGâŠ
> > > >> > >>>>>> but
> > > >> > >>>>>>>> user
> > > >> > >>>>>>>>>>>> facing
> > > >> > >>>>>>>>>>>>>>>>>>> it
> > > >> > >>>>>>>>>>>>>>>>>>>>>> is a
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
> > > >> > >>> Hatter <
> > > >> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > > >> > >>>>>>>>> ryan.hatter@astronomer.io
> > > >> > >>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
> > > >> > >>> swords... at
> > > >> > >>>>>>> least
> > > >> > >>>>>>>>> for
> > > >> > >>>>>>>>>>>> now.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
> > > >> > >>> meaning to
> > > >> > >>>>>>>> Airflow
> > > >> > >>>>>>>>>>>> users.
> > > >> > >>>>>>>>>>>>>>>>>>>>>> Indeed,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> has
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
> > > >> > >>>>>> mathematicians
> > > >> > >>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>> software
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG
> actually
> > > >> > >>>>>> matter.
> > > >> > >>>>>>>> For
> > > >> > >>>>>>>>>>>> someone
> > > >> > >>>>>>>>>>>>>>>>>>>>> new
> > > >> > >>>>>>>>>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
> > > >> > >>>>> orchestration,
> > > >> > >>>>>>> one
> > > >> > >>>>>>>>> of
> > > >> > >>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>> first
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
> > > >> > >>> DAG?"
> > > >> > >>>>>> The
> > > >> > >>>>>>>>> answer
> > > >> > >>>>>>>>>>> is
> > > >> > >>>>>>>>>>>>>>>>>>>>> almost
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
> > > >> > >>> graph.
> > > >> > >>>>> You
> > > >> > >>>>>>>> don't
> > > >> > >>>>>>>>>>> need
> > > >> > >>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>> worry
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
> > > >> > >>>>> workflow."
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for
> at
> > > >> > >>>>> least
> > > >> > >>>>>> a
> > > >> > >>>>>>>>> couple
> > > >> > >>>>>>>>>>>>>>>>>>>>> important
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
> > > >> > >>> mentioned
> > > >> > >>>>>>> above,
> > > >> > >>>>>>>>> "DAG"
> > > >> > >>>>>>>>>>>> is
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
> > > >> > >>> confusing.
> > > >> > >>>>>> We
> > > >> > >>>>>>>> want
> > > >> > >>>>>>>>>>>> Airflow
> > > >> > >>>>>>>>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>> be
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
> > > >> > >>> off the
> > > >> > >>>>>> bat
> > > >> > >>>>>>>>> creates
> > > >> > >>>>>>>>>>>> an
> > > >> > >>>>>>>>>>>>>>>>>>>>>> initial
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and
> Workflow
> > > >> > >>>>>>> Concepts*:
> > > >> > >>>>>>>>> The
> > > >> > >>>>>>>>>>> DAG
> > > >> > >>>>>>>>>>>>>>>>>>> is
> > > >> > >>>>>>>>>>>>>>>>>>>>>> just
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
> > > >> > >>> workflow. The
> > > >> > >>>>>>>>> workflow
> > > >> > >>>>>>>>>>>>>>>>>>> includes
> > > >> > >>>>>>>>>>>>>>>>>>>>>> its
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a
> dozen
> > > >> > >>>>> other
> > > >> > >>>>>>>>>>> parameters,
> > > >> > >>>>>>>>>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>>>>> other
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
> > > >> > >>> Airflow
> > > >> > >>>>>>>> homepage
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > >> > >>>>>>>>>>> https://airflow.apache.org/>>.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
> > > >> > >>> created by
> > > >> > >>>>>> the
> > > >> > >>>>>>>>>>> community
> > > >> > >>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
> > > >> > >>> workflows.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
> > > >> > >>>>> Airflow?"
> > > >> > >>>>>>> docs
> > > >> > >>>>>>>>> page
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>
> > > >> > >>>
> > https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > > <
> > > >> > >>>>>>>>>>>
> > > >> > >>>>>
> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > > >> > >>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
> > > >> > >>> Airflow is
> > > >> > >>>>>>>> without
> > > >> > >>>>>>>>>>> using
> > > >> > >>>>>>>>>>>>>>>>>>>>> "DAG."
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
> > > >> > >>> term
> > > >> > >>>>> is
> > > >> > >>>>>>>>>>> introduced
> > > >> > >>>>>>>>>>>>>>>>>>> out
> > > >> > >>>>>>>>>>>>>>>>>>>>> of
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
> > > >> > >>> explain
> > > >> > >>>>> it:
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> > > >> > >>>>> collection
> > > >> > >>>>>>> of
> > > >> > >>>>>>>>> tasks
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
> > > >> > >>> in
> > > >> > >>>>>> these
> > > >> > >>>>>>>>>>>>>>>>>>> introductions
> > > >> > >>>>>>>>>>>>>>>>>>>>> to
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> > > >> > >>>>>> orchestrate
> > > >> > >>>>>>>>> DAGs;
> > > >> > >>>>>>>>>>> it
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> > > >> > >>>>>> irrelevant
> > > >> > >>>>>>>> to
> > > >> > >>>>>>>>>>> almost
> > > >> > >>>>>>>>>>>>>>>>>>>>> every
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
> > > >> > >>> alias
> > > >> > >>>>> for
> > > >> > >>>>>>> the
> > > >> > >>>>>>>>> term
> > > >> > >>>>>>>>>>>> "DAG"
> > > >> > >>>>>>>>>>>>>>>>>>>>> and
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
> > > >> > >>> "DAG"
> > > >> > >>>>>> with
> > > >> > >>>>>>>>>>>> "workflow".
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start
> running
> > > >> > >>>>> away.
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>
> > > >> > >>>
> > ------------------------------------------------------------------
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > >> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > >> > >>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > >> > >>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
> > > >> > >>>>>>>>>>> dev-help@airflow.apache.org>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >> > >>>>>>>>>>>>>>>>>>>>>
> > > >> > >
> > > >> > >
> > > >> > >
> > > ---------------------------------------------------------------------
> > > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > ---------------------------------------------------------------------
> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
I love "Dynamic Airflow Graph" +10 to that Constance.
And yes. I fully agree with your "somewhat" acyclic statement. I saw people
mentioning "event-driven" workflows to better describe the AI/ML, but that
for me is a synonym of "completely unmanageable workflows" (similar to
micro-services) - what I was alluding to was exactly this - something like
curren DAG with parts of it being potentially cyclic - so we are completely
aligned here :). And that's what make it "not DAG - in mathematical sense"
- even more reason to depart from it,
J.
On Tue, Feb 18, 2025 at 5:23âŻPM Constance Martineau
<co...@astronomer.io.invalid> wrote:
> Hey all,
>
> I appreciate the discussion about how we position the concept of a DAG in
> Airflow. While I agree with updating documentation to de-emphasize the
> strict mathematical definition, I do not think we should change the public
> interface to use "dag" instead of "DAG" everywhere. It adds
> unnecessary churn and breaks consistency across the ecosystem.
>
> Regarding the broader discussion on whether DAGs in Airflow should remain
> acyclic: I acknowledge that some workflows, particularly in ML and AI,
> require cyclical behaviour (example: iterative refinement until a condition
> is met). However, I think it's important to recognize that even in those
> cases, the workflow itself still has an overall start and end. It's not an
> unbounded loop but rather a controlled execution process that happens to
> have cycles *within* it.
>
> That being said, I do see value in reinterpreting "DAG" to better align
> with how Airflow is actually used, and love shifting away from "Directed
> Acyclic Graph" in favour of something more intuitive. However "Data Airflow
> Graph" makes it sound like we're defining Airflow itself rather than
> describing the structure of workflows within it.
>
> If we want a new meaning for DAG that resonates with our base while keeping
> Airflow's identity clear, I am proposing three alternatives:
>
>
> 1. *Dynamic Airflow Graph*: Ties directly to Airflow while emphasizing
> flexibility
> 2. *Distributed Automation Graph*: Highlights execution across systems
> and automations without tying it strictly to data
> 3. *Directed Activity Graph*: Retains the useful meaning of "Directed"
> while shifting away from strict acyclic constraints. (My personal
> favourite)
>
> I think any shift in terminology should be driven by what best communicates
> Airflow's role workflow automation, rather than just distancing ourselves
> from the mathematical definition.
>
> Curious to hear more thoughts!
>
> On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Sorry for the few typos ... My slight dyslexia did not help and
> autocorrect
> > did not help either this time : dept -> debt, routed -> rooted and a few
> > other small typos :) .
> >
> > On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > First of all - yes, agree with Lee and Ash, I think we do not need to
> > > change the public interface, I'd also be -0 on that, but docs, internal
> > > strings, yes we could.
> > >
> > > But .... After thinking a bit and looking at the discussion here, where
> > > people have concerns, I have an alternative take. And I know this might
> > be
> > > controversial, and some people might find it confusing, but why don't
> we
> > > come up with a new acronym for DAG, one that will be more airflow and
> > less
> > > "mathematics/algorithm". And I have, I think a good reason for it, and
> I
> > > even have a proposal. Bear with me,
> > >
> > > DAG => *Data Airflow Graph*
> > >
> > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now (with
> > > exception of some niche products - you know which one I speak about -
> > which
> > > are unfortunate enough to use the "Dag" in their name - I will explain
> > why
> > > this is unfortunate, IMHO).
> > >
> > > We are lucky to have Airflow for "A" in the acronym and we can make it
> > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow.
> > >
> > > Now, why I think "Directed Acyclic Graph" is unfortunate.
> > >
> > > Not only because "Directed" and "Acyclic" are mostly not in the
> > vocabulary
> > > of most of our users - as Ryan, starting this thread explained yes, I
> > agree
> > > with it, but also I think there is a more important reason - I simply
> > > believe in the future our DAGs might not (and likely will not) be
> > "acyclic".
> > >
> > > I know for some it might be herezy and we have not discussed any of
> that
> > > yet, but more and more I keep on hearing that Machine Learning
> Workflows
> > of
> > > the future are often somewhat cyclic. In our case there is one type of
> > > repeat-in-cycle behaviour - we have "task retry on failure" - but this
> is
> > > only on failure and it basically mens "idempotent" retry - we want to
> > redo
> > > something and get single result out of that and basically discard the
> > > previous result. But machine learning/AI workflows are different. I
> had a
> > > lot of conversations recently (I even co-organized "AI low-level
> > > engineering and hacking" devroom to learn more about those) - and I
> keep
> > on
> > > hearing that "Repeat a sequence of things several times and refine" is
> > > precisely what AI / ML workflows will need - especially with Agentic
> > > workflows - often we have cooperating agents which **could** be part of
> > the
> > > same DAG, but currently in Airflow where our DAGs are "Acyclic" we can
> > > only express "repeat several time and refine" as a single task that
> will
> > do
> > > such refining. If we would like to split such an "agent" doing their
> own
> > > work in "cycles" and communicating with other "agents" doing the same -
> > > represented as a multi-task-entity, we have no way to express it using
> > our
> > > current "acyclic" semantics.
> > >
> > > But - other than our implementation makes that assumption currently -
> > > there is nothing fundamental with Airflow being able to run such
> > "partially
> > > cyclic workflows". We have all the pieces, we would have to change
> > > dependency calculation to allow for cycles, update UI to handle cycles
> > > representation - and we could in theory quite seamlessly, support
> > "cyclic"
> > > workflows.
> > >
> > > I know, I know, that's the first time we hear it and that's huge
> change -
> > > of course, I do not want to start discussing any details of it, not
> > before
> > > Airflow 3.0 is out - but I think we should be open that at some point
> of
> > > time the "Acyclic" property is going to go away, And my personal
> > intuition
> > > tells me that a year from now we will have acyclic workflows (assuming
> > that
> > > all the foundational work we do on Airflow 3 will really help in
> > decreasing
> > > all the technical dept, and will open our minds to new ideas).
> > >
> > > That's why I think - the sooner we get rid of "Directed Acyclic" - the
> > > better. that will - even for us, maintainer free us from being routed
> in
> > > the acyclic nature of Airflow.
> > >
> > > How about we change to "Data Airflow Graph" (or any other good acronym
> > > which has (A)irflow instead of (A)cyclic). The name is just a proposal.
> > >
> > > Now.. I go hide somewhere as I believe that might cause a "flamewar"
> :),
> > > unless on request of Ryan you already sheathed your swords in the
> > > discussion (which seems everyone did).
> > >
> > > But honestly - I would really love what you think of "let's just break
> > > with Acyclic NOW" and shape our future by using a different acronym.
> > >
> > > J.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > > ephraimanierobi@gmail.com> wrote:
> > >
> > >> When I started contributing to Airflow, I had to read up on DAG, and
> > now I
> > >> know what it means. I wonder if we are about to have users who use the
> > >> term
> > >> Dag without knowing that there's a DAG.
> > >>
> > >> Slightly concerned.
> > >>
> > >> -ephraim
> > >>
> > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com> wrote:
> > >>
> > >> > For user-facing things (e.g., `from airflow import DAG`), Iâm more
> > like
> > >> > -0. But for documentation, docstring, internal things and etc., we
> > >> probably
> > >> > could still change most of them?
> > >> >
> > >> > Best,
> > >> > Wei
> > >> >
> > >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org>
> > >> wrote:
> > >> > >
> > >> > > I also donât personally think itâs worth the pain (not to mention
> > >> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
> > >> > >
> > >> > > -ash
> > >> > >
> > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> > daniel.imberman@gmail.com
> > >> >
> > >> > wrote:
> > >> > >>
> > >> > >> I think my biggest concern is a marketing one and not a technical
> > >> one.
> > >> > >>
> > >> > >> As has been mentioned on the thread the terms airflow and dag are
> > >> kind
> > >> > of
> > >> > >> synonymous and I certainly donât want to give the impression that
> > we
> > >> are
> > >> > >> breaking more than we are breaking.
> > >> > >>
> > >> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> > >> > >>
> > >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org>
> wrote:
> > >> > >>
> > >> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff rules
> > are
> > >> > best
> > >> > >>> suited for user-facing things but not the airflow code base
> > itself.
> > >> If
> > >> > what
> > >> > >>> we mean is adding a rule to avoid users using "DAG" *after* we
> > >> rename
> > >> > it,
> > >> > >>> it's definitely a +1000.
> > >> > >>>
> > >> > >>> I just created GitHub issues for this removing "DAG"
> > >> > >>>
> > >> > >>> https://github.com/apache/airflow/issues/46842
> > >> > >>> https://github.com/apache/airflow/issues/46843
> > >> > >>>
> > >> > >>> One thing I'm not sure about is whether we want to get rid of
> > `from
> > >> > >>> airflow import DAG` as well. đ€
> > >> > >>>
> > >> > >>> Best,
> > >> > >>> Wei
> > >> > >>>
> > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > >> > >>>> Hard to say until it's looked at :)
> > >> > >>>>
> > >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> > >> aritrabasu1999@gmail.com
> > >> > >
> > >> > >>>> wrote:
> > >> > >>>>
> > >> > >>>>> I can take it up, it's mostly just a doc update right? Or are
> we
> > >> > doing
> > >> > >>> code
> > >> > >>>>> files replacement too?
> > >> > >>>>> --
> > >> > >>>>> Regards,
> > >> > >>>>> Aritra Basu
> > >> > >>>>>
> > >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <jarek@potiuk.com
> >
> > >> > wrote:
> > >> > >>>>>
> > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any
> > >> volunteers
> > >> > >>> to
> > >> > >>>>>> make it ? It's fun.
> > >> > >>>>>>
> > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> > droiddev510e@gmail.com>
> > >> > >>> wrote:
> > >> > >>>>>>
> > >> > >>>>>>> +1 for ruff rules :)
> > >> > >>>>>>>
> > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in
> the
> > >> > >>> Airflow
> > >> > >>>>>>> docs, in line with the new UI changes and to make the
> renaming
> > >> > >>>>>>> consistent across user-facing pages.
> > >> > >>>>>>>
> > >> > >>>>>>> Regards,
> > >> > >>>>>>> Omkar
> > >> > >>>>>>>
> > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> > jarek@potiuk.com>
> > >> > >>> wrote:
> > >> > >>>>>>>
> > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> > >> > >>>>>>>>
> > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <weilee@apache.org
> >
> > >> > >>> wrote:
> > >> > >>>>>>>>
> > >> > >>>>>>>>> It seems that our current conclusion is to use "dag" or
> > "Dag"
> > >> > >>>>> instead
> > >> > >>>>>>> of
> > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in
> the
> > >> > >>>>> codebase
> > >> > >>>>>>> with
> > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might
> be
> > >> > >>> đ€),
> > >> > >>>>>>> should
> > >> > >>>>>>>> we
> > >> > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> Best,
> > >> > >>>>>>>>> Wei
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
> > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > >> > >>>>>>>>>> Let me know if that's the direction we want to go.
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
> > >> > >>> synonymous
> > >> > >>>>>> with
> > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and explain
> > >> > >>> what
> > >> > >>>>>>> Airflow
> > >> > >>>>>>>>> is
> > >> > >>>>>>>>>>> all about. Although folks would ask, I donât ever use
> the
> > >> > >>>>>>>> mathematical
> > >> > >>>>>>>>>>> definition of DAG.
> > >> > >>>>>>>>>>> It will be challenging and possibly confusing for many
> > >> > >>> users
> > >> > >>>>>> making
> > >> > >>>>>>>>> such a
> > >> > >>>>>>>>>>> change - I would rather direct that energy to appending
> > the
> > >> > >>>>>> Oxford
> > >> > >>>>>>>>>>> definition of "dag" to include a reference to workflows
> in
> > >> > >>>>>> Airflow.
> > >> > >>>>>>>> __
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> -- Rajesh
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <
> jarek@potiuk.com
> > >> > >>>>>> <mailto:
> > >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the
> > >> > >>>>> organization.
> > >> > >>>>>> Do
> > >> > >>>>>>>> not
> > >> > >>>>>>>>>>> click links or open attachments unless you can confirm
> the
> > >> > >>>>> sender
> > >> > >>>>>>> and
> > >> > >>>>>>>>> know
> > >> > >>>>>>>>>>> the content is safe.
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > >> > >>>>> expéditeur
> > >> > >>>>>>>>> externe.
> > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> jointe
> > >> > >>> si
> > >> > >>>>> vous
> > >> > >>>>>>> ne
> > >> > >>>>>>>>> pouvez
> > >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous
> nâĂȘtes
> > >> > >>> pas
> > >> > >>>>>>>> certain
> > >> > >>>>>>>>> que
> > >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> From Guido's post:
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> "Around this time the renaming seems to have been
> > renamed".
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> Naming is hard.
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> J.
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > >> > >>>>> droiddev510e@gmail.com
> > >> > >>>>>>>>> <mailto:
> > >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an
> > >> > >>> Airflow
> > >> > >>>>>> term
> > >> > >>>>>>>> now
> > >> > >>>>>>>>>>>> rather
> > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or
> not,
> > >> > >>> I've
> > >> > >>>>>>>> worked
> > >> > >>>>>>>>> with
> > >> > >>>>>>>>>>>> work
> > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now
> > >> > >>> and are
> > >> > >>>>>> pro
> > >> > >>>>>>>>> devs
> > >> > >>>>>>>>>>> but
> > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For
> > >> > >>> them,
> > >> > >>>>> dag
> > >> > >>>>>> =
> > >> > >>>>>>>>> Airflow.
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
> > >> > >>> will be
> > >> > >>>>> a
> > >> > >>>>>>>> major
> > >> > >>>>>>>>>>>> behavior
> > >> > >>>>>>>>>>>> change for the developer community, so we'll need a
> solid
> > >> > >>>>> plan
> > >> > >>>>>> on
> > >> > >>>>>>>>> how to
> > >> > >>>>>>>>>>>> introduce it, when we do.
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the
> Great
> > >> > >>>>>> Renaming
> > >> > >>>>>>>> in
> > >> > >>>>>>>>> core
> > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably
> > >> > >>> get
> > >> > >>>>>> some
> > >> > >>>>>>>>>>> learnings
> > >> > >>>>>>>>>>>> from there? Who knows!
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>> Guido's blog (2009):
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>
> > >> > >>>
> > >> >
> > >>
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > >> > >>>>>>>>>>> <
> > >> > >>>>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>
> > >> > >>>
> > >> >
> > >>
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>>>> Follow-up discussion (2024):
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>
> > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > >> > >>>>>>> <
> > >> > >>>>>>>>>>>
> > >> > >>>>>
> https://discuss.python.org/t/finishing-the-great-renaming/54082
> > >
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>> Cheers,
> > >> > >>>>>>>>>>>> Omkar
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > >> > >>> weilee.rx@gmail.com
> > >> > >>>>>>>>> <mailto:
> > >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>>> I think we should probably just accept it as an
> airflow
> > >> > >>>>> term.
> > >> > >>>>>>> At
> > >> > >>>>>>>>> least,
> > >> > >>>>>>>>>>>>> thatâs how I understood it when I first used Airflow.
> I
> > >> > >>>>> feel
> > >> > >>>>>>>>> renaming
> > >> > >>>>>>>>>>> it
> > >> > >>>>>>>>>>>> at
> > >> > >>>>>>>>>>>>> this stage would require considerable effort from
> > >> > >>>>> maintainers
> > >> > >>>>>>> and
> > >> > >>>>>>>>>>>> existing
> > >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > >> > >>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>> Best,
> > >> > >>>>>>>>>>>>> Wei
> > >> > >>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > >> > >>>>>> kaxilnaik@gmail.com
> > >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > >> > >>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
> > >> > >>>>> this
> > >> > >>>>>>>>> discussion
> > >> > >>>>>>>>>>>> for
> > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
> > >> > >>>>>>> disruption
> > >> > >>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > >> > >>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
> > >> > >>>>>> experience
> > >> > >>>>>>>>> what a
> > >> > >>>>>>>>>>>> dag
> > >> > >>>>>>>>>>>>> is,
> > >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes -
> > >> > >>> stuff
> > >> > >>>>>> like
> > >> > >>>>>>>>> dags
> > >> > >>>>>>>>>>>> being
> > >> > >>>>>>>>>>>>> a
> > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
> > >> > >>>>>> doesn't
> > >> > >>>>>>>>> come up.
> > >> > >>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> > >> > >>> embrace
> > >> > >>>>> the
> > >> > >>>>>>> name
> > >> > >>>>>>>>> and
> > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
> > >> > >>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > >> > >>>>> vikram@astronomer.io.inva
> > >> > >>>>>>>>> lid>
> > >> > >>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> > >> > >>>>> struggling
> > >> > >>>>>>> with
> > >> > >>>>>>>>> this
> > >> > >>>>>>>>>>>>> when I
> > >> > >>>>>>>>>>>>>>>> started working with Airflow.
> > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
> > >> > >>>>>>>> established
> > >> > >>>>>>>>>>>> concept
> > >> > >>>>>>>>>>>>>>> now
> > >> > >>>>>>>>>>>>>>>> regardless of the origin.
> > >> > >>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
> > >> > >>> best
> > >> > >>>>>>>> expressed
> > >> > >>>>>>>>> by
> > >> > >>>>>>>>>>>>> Daniel
> > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
> > >> > >>> than
> > >> > >>>>>> the
> > >> > >>>>>>>>> computer
> > >> > >>>>>>>>>>>>>>> science
> > >> > >>>>>>>>>>>>>>>> concept.
> > >> > >>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>> Best regards,
> > >> > >>>>>>>>>>>>>>>> Vikram
> > >> > >>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > >> > >>>>>> onikolas@amazon.com.inva
> > >> > >>>>>>>>> lid
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
> > >> > >>> right
> > >> > >>>>>> Ryan,
> > >> > >>>>>>>> DAG
> > >> > >>>>>>>>>>> isn't
> > >> > >>>>>>>>>>>>>>> great
> > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
> > >> > >>> cause
> > >> > >>>>> much
> > >> > >>>>>>>> more
> > >> > >>>>>>>>>>>> wreckage
> > >> > >>>>>>>>>>>>>>>> than
> > >> > >>>>>>>>>>>>>>>>> it solves.
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
> > >> > >>>>> defining
> > >> > >>>>>>>> DAG.
> > >> > >>>>>>>>> I
> > >> > >>>>>>>>>>>> think
> > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
> > >> > >>> for a
> > >> > >>>>>> while
> > >> > >>>>>>>> now
> > >> > >>>>>>>>>>>> anyway,
> > >> > >>>>>>>>>>>>>>> so
> > >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> Cheers,
> > >> > >>>>>>>>>>>>>>>>> Niko
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> ________________________________
> > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > >> > >>>>>>>>> ash@apache.org>>
> > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > >> > >>>>>> dev@airflow.apache.org
> > >> > >>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
> > >> > >>> term
> > >> > >>>>>> "DAG"
> > >> > >>>>>>>>> for end
> > >> > >>>>>>>>>>>>>>> users
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
> > >> > >>>>>>>>> organization.
> > >> > >>>>>>>>>>> Do
> > >> > >>>>>>>>>>>>> not
> > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can
> > >> > >>> confirm
> > >> > >>>>>> the
> > >> > >>>>>>>>> sender
> > >> > >>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>> know
> > >> > >>>>>>>>>>>>>>>>> the content is safe.
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> > >> > >>> dâun
> > >> > >>>>>>>>> expéditeur
> > >> > >>>>>>>>>>>>>>> externe.
> > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > >> > >>>>>> jointe
> > >> > >>>>>>> si
> > >> > >>>>>>>>> vous
> > >> > >>>>>>>>>>> ne
> > >> > >>>>>>>>>>>>>>>> pouvez
> > >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> > >> > >>> vous
> > >> > >>>>>> nâĂȘtes
> > >> > >>>>>>>> pas
> > >> > >>>>>>>>>>>> certain
> > >> > >>>>>>>>>>>>>>>> que
> > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
> > >> > >>> term â
> > >> > >>>>>>>> getting
> > >> > >>>>>>>>> to
> > >> > >>>>>>>>>>>>> re-use
> > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
> > >> > >>>>> sides
> > >> > >>>>>>> have
> > >> > >>>>>>>>> good
> > >> > >>>>>>>>>>>>>>> reasons
> > >> > >>>>>>>>>>>>>>>>> presented.
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>> -a
> > >> > >>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
> > >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> > >> > >>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
> > >> > >>> from,
> > >> > >>>>>>> "well,
> > >> > >>>>>>>>> no
> > >> > >>>>>>>>>>> one
> > >> > >>>>>>>>>>>>>>>>> actually
> > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> > >> > >>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > >> > >>>>>>>>> jarek@potiuk.com
> > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > >> > >>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
> > >> > >>> the
> > >> > >>>>>>>>>>> tongue-twister,
> > >> > >>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > >> > >>>>>>>>> jarek@potiuk.com
> > >> > >>>>>>>>>>> <ma...@potiuk.com>>
> > >> > >>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
> > >> > >>> would
> > >> > >>>>> very
> > >> > >>>>>>>> much
> > >> > >>>>>>>>> want
> > >> > >>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>> be
> > >> > >>>>>>>>>>>>>>>>>>> able
> > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> > >> > >>> further.
> > >> > >>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
> > >> > >>> it's a
> > >> > >>>>>>>>>>>> tongue-twister,
> > >> > >>>>>>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>> I
> > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> > >> > >>> pronounce
> > >> > >>>>>> it
> > >> > >>>>>>>>>>> properly.
> > >> > >>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>> J.
> > >> > >>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > >> > >>>>>>>> brent@astronomer.io.inva
> > >> > >>>>>>>>>> lid>
> > >> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
> > >> > >>>>> starting
> > >> > >>>>>> on
> > >> > >>>>>>>>> AIP-38
> > >> > >>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>> modernize
> > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > >> > >>>>>>>>> descriptive of
> > >> > >>>>>>>>>>>> what
> > >> > >>>>>>>>>>>>>>>> one
> > >> > >>>>>>>>>>>>>>>>>>> is
> > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
> > >> > >>> is an
> > >> > >>>>>>>>>>> implementation
> > >> > >>>>>>>>>>>>>>>>> detail.
> > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
> > >> > >>> point
> > >> > >>>>>> "DAG"
> > >> > >>>>>>>> has
> > >> > >>>>>>>>>>> become
> > >> > >>>>>>>>>>>>>>>> "dag"
> > >> > >>>>>>>>>>>>>>>>>>> , a
> > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > >> > >>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
> > >> > >>>>>> discussion,
> > >> > >>>>>>>>> Airflow
> > >> > >>>>>>>>>>>>>>>> Summit
> > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
> > >> > >>>>>> embrace
> > >> > >>>>>>>>> "dag".
> > >> > >>>>>>>>>>> A
> > >> > >>>>>>>>>>>>>>> user
> > >> > >>>>>>>>>>>>>>>>>>> just
> > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
> > >> > >>>>> concept
> > >> > >>>>>>>>> behind
> > >> > >>>>>>>>>>> that
> > >> > >>>>>>>>>>>>>>>>> word. I
> > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > >> > >>> refactoring so
> > >> > >>>>>>> much
> > >> > >>>>>>>>> code,
> > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
> > >> > >>>>> questions,
> > >> > >>>>>>>> etc.
> > >> > >>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> > >> > >>> Standish
> > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
> > >> > >>> of
> > >> > >>>>>> pain
> > >> > >>>>>>>> for
> > >> > >>>>>>>>>>>>>>>>> questionable
> > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
> > >> > >>> idea. I
> > >> > >>>>>>> agree
> > >> > >>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>> association
> > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
> > >> > >>>>>>> invasive
> > >> > >>>>>>>>>>>>>>> mitigations.
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
> > >> > >>>>>> de-emphasize
> > >> > >>>>>>>> the
> > >> > >>>>>>>>>>> math
> > >> > >>>>>>>>>>>>>>> nerd
> > >> > >>>>>>>>>>>>>>>>>>>>> origins
> > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> > >> > >>> website /
> > >> > >>>>>> etc,
> > >> > >>>>>>>>> *never
> > >> > >>>>>>>>>>>>>>>> define*
> > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
> > >> > >>>>>> graph.
> > >> > >>>>>>>>> Always
> > >> > >>>>>>>>>>>>>>> define
> > >> > >>>>>>>>>>>>>>>>> it
> > >> > >>>>>>>>>>>>>>>>>>>>> as a
> > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
> > >> > >>> like a
> > >> > >>>>>>>>> historical
> > >> > >>>>>>>>>>>>>>>>> footnote,
> > >> > >>>>>>>>>>>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
> > >> > >>> hidden.
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
> > >> > >>> docs
> > >> > >>>>> in
> > >> > >>>>>>>>> general
> > >> > >>>>>>>>>>> e.g.
> > >> > >>>>>>>>>>>>>>>>>>> writing
> > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> > >> > >>> "DAGs"
> > >> > >>>>>> etc.
> > >> > >>>>>>>> The
> > >> > >>>>>>>>>>>> upper
> > >> > >>>>>>>>>>>>>>>> case
> > >> > >>>>>>>>>>>>>>>>>>>>> part
> > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
> > >> > >>> "dag" in
> > >> > >>>>>>>>> airlfow is
> > >> > >>>>>>>>>>>> just
> > >> > >>>>>>>>>>>>>>>> an
> > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
> > >> > >>> "DAGs"
> > >> > >>>>> is
> > >> > >>>>>>> not
> > >> > >>>>>>>>> really
> > >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
> > >> > >>> is
> > >> > >>>>> its
> > >> > >>>>>>> own
> > >> > >>>>>>>>>>> thing,
> > >> > >>>>>>>>>>>> is
> > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
> > >> > >>> nobody
> > >> > >>>>>> knows
> > >> > >>>>>>>>> about
> > >> > >>>>>>>>>>>>>>>> anyway),
> > >> > >>>>>>>>>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
> > >> > >>> normal
> > >> > >>>>>>> people
> > >> > >>>>>>>>>>>>>>> understand.
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
> > >> > >>> <
> > >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > >> > >>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
> > >> > >>> will
> > >> > >>>>> be
> > >> > >>>>>>>>> extremely
> > >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
> > >> > >>> will
> > >> > >>>>>> make
> > >> > >>>>>>> a
> > >> > >>>>>>>>> lot of
> > >> > >>>>>>>>>>>>>>>>>>> "google"
> > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
> > >> > >>>>> finding
> > >> > >>>>>>> the
> > >> > >>>>>>>>> right
> > >> > >>>>>>>>>>>>>>>>>>> answers.
> > >> > >>>>>>>>>>>>>>>>>>>>>> This
> > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> > >> > >>> besides the
> > >> > >>>>>>>>> community
> > >> > >>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>> ideas
> > >> > >>>>>>>>>>>>>>>>>>>>> that
> > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
> > >> > >>>>> examples,
> > >> > >>>>>>>>> problems
> > >> > >>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>>>> solutions
> > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> > >> > >>> remember
> > >> > >>>>>> that
> > >> > >>>>>>>>> all the
> > >> > >>>>>>>>>>>> AI
> > >> > >>>>>>>>>>>>>>>>>>>>> trained
> > >> > >>>>>>>>>>>>>>>>>>>>>> on
> > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> > >> > >>> matching
> > >> > >>>>>>> queries
> > >> > >>>>>>>> of
> > >> > >>>>>>>>>>>> people.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
> > >> > >>>>>> switch.
> > >> > >>>>>>>>> And if
> > >> > >>>>>>>>>>> we
> > >> > >>>>>>>>>>>>>>> do
> > >> > >>>>>>>>>>>>>>>> -
> > >> > >>>>>>>>>>>>>>>>>>> I
> > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
> > >> > >>> instead
> > >> > >>>>> of
> > >> > >>>>>>>>> `dag` if
> > >> > >>>>>>>>>>>> not
> > >> > >>>>>>>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>> above
> > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
> > >> > >>> that it
> > >> > >>>>>>> might
> > >> > >>>>>>>>> cause
> > >> > >>>>>>>>>>>> more
> > >> > >>>>>>>>>>>>>>>>>>>>> problems
> > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
> > >> > >>> think
> > >> > >>>>>>> it's
> > >> > >>>>>>>> a
> > >> > >>>>>>>>> good
> > >> > >>>>>>>>>>>>>>>> idea, I
> > >> > >>>>>>>>>>>>>>>>>>>>> am
> > >> > >>>>>>>>>>>>>>>>>>>>>> ok
> > >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> > >> > >>> Bhakat
> > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> > >> > >>> <mailto:
> > >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
> > >> > >>> sense
> > >> > >>>>> to
> > >> > >>>>>>>>> someone
> > >> > >>>>>>>>>>> new
> > >> > >>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>> workflow
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
> > >> > >>> nature
> > >> > >>>>>> of
> > >> > >>>>>>>>> being
> > >> > >>>>>>>>>>>>>>> acyclic.
> > >> > >>>>>>>>>>>>>>>>>>>>> Sure,
> > >> > >>>>>>>>>>>>>>>>>>>>>>> as
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> > >> > >>> workaround it.
> > >> > >>>>>>>> Still,
> > >> > >>>>>>>>> in
> > >> > >>>>>>>>>>> my
> > >> > >>>>>>>>>>>>>>>>>>>>> opinion,
> > >> > >>>>>>>>>>>>>>>>>>>>>>> there
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
> > >> > >>>>>>> workflow
> > >> > >>>>>>>>>>>>>>>>>>> orchestration.
> > >> > >>>>>>>>>>>>>>>>>>>>>> Most
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
> > >> > >>>>> covered
> > >> > >>>>>>>>> using an
> > >> > >>>>>>>>>>>>>>>> acyclic
> > >> > >>>>>>>>>>>>>>>>>>>>>> manner
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> > >> > >>> idempotency. So
> > >> > >>>>> I
> > >> > >>>>>>>> would
> > >> > >>>>>>>>> want
> > >> > >>>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > >> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> > >> > >>> bernd.stroehle@kosakya.de>>
> > >> > >>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
> > >> > >>>>> Airflow
> > >> > >>>>>>> Fan;
> > >> > >>>>>>>>> so
> > >> > >>>>>>>>>>> many
> > >> > >>>>>>>>>>>>>>> new
> > >> > >>>>>>>>>>>>>>>>>>>>>> ideas.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
> > >> > >>>>>> replaced
> > >> > >>>>>>> by
> > >> > >>>>>>>>> the
> > >> > >>>>>>>>>>>> more
> > >> > >>>>>>>>>>>>>>>>>>>>> general
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> > >> > >>>>>> (Petri)
> > >> > >>>>>>>> Net
> > >> > >>>>>>>>>>> (AFN)
> > >> > >>>>>>>>>>>>>>>>>>> (maybe
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> without
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
> > >> > >>>>> Graph
> > >> > >>>>>>>>> Database.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> > >> > >>> Graph
> > >> > >>>>>> (AFG)
> > >> > >>>>>>>>> might be
> > >> > >>>>>>>>>>>>>>>>>>>>> assigned
> > >> > >>>>>>>>>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>>> an
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
> > >> > >>> of a
> > >> > >>>>>>>> library.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
> > >> > >>>>> structure
> > >> > >>>>>>>> than
> > >> > >>>>>>>>> a
> > >> > >>>>>>>>>>>>>>>>>>>>> configuration
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > >> > >>>>>>>>>>> bernd.stroehle@gmail.com>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> > >> > >>> <ikholopov@google.com.INVA
> > >> > >>>>>>>>> <mailto:
> > >> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > >> > >>>>>>>>> dev@airflow.apache.org
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
> > >> > >>> term
> > >> > >>>>>>> "DAG"
> > >> > >>>>>>>>> for
> > >> > >>>>>>>>>>> end
> > >> > >>>>>>>>>>>>>>>>>>> users
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
> > >> > >>>>>> suboptimal,
> > >> > >>>>>>> it
> > >> > >>>>>>>>> is
> > >> > >>>>>>>>>>> part
> > >> > >>>>>>>>>>>>>>> of
> > >> > >>>>>>>>>>>>>>>>>>>>>> Airflow
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
> > >> > >>> levels,
> > >> > >>>>> that
> > >> > >>>>>>> any
> > >> > >>>>>>>>>>> attempt
> > >> > >>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>> change
> > >> > >>>>>>>>>>>>>>>>>>>>>>> it
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
> > >> > >>> reduce it.
> > >> > >>>>>> The
> > >> > >>>>>>>>> only
> > >> > >>>>>>>>>>>> thing
> > >> > >>>>>>>>>>>>>>>>>>>>> that is
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> worse
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
> > >> > >>> when
> > >> > >>>>>>> there
> > >> > >>>>>>>>> are
> > >> > >>>>>>>>>>> two
> > >> > >>>>>>>>>>>>>>> ways
> > >> > >>>>>>>>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> define
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
> > >> > >>>>> tutorials
> > >> > >>>>>>>> will
> > >> > >>>>>>>>>>>> suddenly
> > >> > >>>>>>>>>>>>>>>>>>>>> become
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
> > >> > >>> as
> > >> > >>>>>>> "DAG"s.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
> > >> > >>> users
> > >> > >>>>> away
> > >> > >>>>>>>> with
> > >> > >>>>>>>>> a
> > >> > >>>>>>>>>>>> number
> > >> > >>>>>>>>>>>>>>>>>>> of
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> breaking
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
> > >> > >>>>>> breaking
> > >> > >>>>>>>>> changes
> > >> > >>>>>>>>>>>> for
> > >> > >>>>>>>>>>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>>> most
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
> > >> > >>> are
> > >> > >>>>>>>> looking
> > >> > >>>>>>>>> for.
> > >> > >>>>>>>>>>>>>>>>>>>>> Attempting
> > >> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
> > >> > >>>>> interpreted
> > >> > >>>>>>> as
> > >> > >>>>>>>> an
> > >> > >>>>>>>>>>> even
> > >> > >>>>>>>>>>>>>>>>>>>>> stronger
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> signal
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
> > >> > >>> the
> > >> > >>>>>> term
> > >> > >>>>>>>> in
> > >> > >>>>>>>>> the
> > >> > >>>>>>>>>>>> long
> > >> > >>>>>>>>>>>>>>>>>>>>> run. I
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> even
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
> > >> > >>> it in
> > >> > >>>>>> the
> > >> > >>>>>>>> DAG
> > >> > >>>>>>>>>>>>>>> definition
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> interface.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
> > >> > >>> other
> > >> > >>>>>> places
> > >> > >>>>>>>> in
> > >> > >>>>>>>>>>>> Airflow,
> > >> > >>>>>>>>>>>>>>>>>>> of
> > >> > >>>>>>>>>>>>>>>>>>>>>> which
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
> > >> > >>> Harenslak
> > >> > >>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > >> > >>>>>>>> bas@astronomer.io.inva
> > >> > >>>>>>>>>> lid
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
> > >> > >>> have
> > >> > >>>>>>> already
> > >> > >>>>>>>>> faded
> > >> > >>>>>>>>>>>> over
> > >> > >>>>>>>>>>>>>>>>>>>>> time,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
> > >> > >>>>>> create
> > >> > >>>>>>>>> cyclic
> > >> > >>>>>>>>>>>>>>>>>>> graphs,
> > >> > >>>>>>>>>>>>>>>>>>>>>> e.g.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
> > >> > >>>>> these
> > >> > >>>>>>>>>>> properties
> > >> > >>>>>>>>>>>>>>>>>>>>>> vanishing
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
> > >> > >>>>>>> perspective I
> > >> > >>>>>>>>>>> support
> > >> > >>>>>>>>>>>>>>>>>>>>>> changing
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
> > >> > >>>>>> naming:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
> > >> > >>> name.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
> > >> > >>> change, I
> > >> > >>>>>>> suggest
> > >> > >>>>>>>>>>>> defining
> > >> > >>>>>>>>>>>>>>> a
> > >> > >>>>>>>>>>>>>>>>>>>>>> clear
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
> > >> > >>> the
> > >> > >>>>>>>>> deprecation in
> > >> > >>>>>>>>>>>>>>>>>>>>> Airflow
> > >> > >>>>>>>>>>>>>>>>>>>>>> 3,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > >> > >>>>>>>> neil4real@gmail.com
> > >> > >>>>>>>>>>> <ma...@gmail.com>>
> > >> > >>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > >> > >>>>>>>> especially
> > >> > >>>>>>>>> when
> > >> > >>>>>>>>>>>>>>>>>>> most
> > >> > >>>>>>>>>>>>>>>>>>>>>> other
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
> > >> > >>> wholeheartedly.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
> > >> > >>>>> confusing
> > >> > >>>>>>>>> about it
> > >> > >>>>>>>>>>> at
> > >> > >>>>>>>>>>>>>>>>>>>>> all,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
> > >> > >>>>>>> confusing
> > >> > >>>>>>>>> to
> > >> > >>>>>>>>>>> most
> > >> > >>>>>>>>>>>>>>>>>>>>> users
> > >> > >>>>>>>>>>>>>>>>>>>>>>> who
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> > >> > >>>>> Chung
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > >> > >>>>>>>>> tp@astronomer.io.inva
> > >> > >>>>>>>>>>>> lid
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
> > >> > >>> the
> > >> > >>>>> term
> > >> > >>>>>>>> DAG.
> > >> > >>>>>>>>> The
> > >> > >>>>>>>>>>>> only
> > >> > >>>>>>>>>>>>>>>>>>>>>>> problem
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
> > >> > >>>>>>> myselfâto
> > >> > >>>>>>>>> stop
> > >> > >>>>>>>>>>>> using
> > >> > >>>>>>>>>>>>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
> > >> > >>> mentioned
> > >> > >>>>>>>>> âworkflowâ is
> > >> > >>>>>>>>>>>> not
> > >> > >>>>>>>>>>>>>>>>>>>>> very
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> > >> > >>>>>>> proposed,
> > >> > >>>>>>>>> but
> > >> > >>>>>>>>>>>> itâs a
> > >> > >>>>>>>>>>>>>>>>>>>>> bit
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
> > >> > >>> places
> > >> > >>>>> and
> > >> > >>>>>>> kept
> > >> > >>>>>>>>>>> finding
> > >> > >>>>>>>>>>>>>>>>>>>>> myself
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
> > >> > >>> wanting to
> > >> > >>>>>>>> clarify
> > >> > >>>>>>>>> âDAG
> > >> > >>>>>>>>>>>>>>>>>>>>>>> definitionâ
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
> > >> > >>> Scheffler
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > >> > >>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
> > >> > >>> exactly
> > >> > >>>>> same
> > >> > >>>>>>>>>>>> observation,
> > >> > >>>>>>>>>>>>>>>>>>>>> had a
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
> > >> > >>> always
> > >> > >>>>> an
> > >> > >>>>>>>>>>> introduction
> > >> > >>>>>>>>>>>>>>>>>>> if
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
> > >> > >>> makes
> > >> > >>>>>>> sense.
> > >> > >>>>>>>>>>>>>>>>>>> Especially
> > >> > >>>>>>>>>>>>>>>>>>>>>> when
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> also
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> > >> > >>>>>>> reasonable
> > >> > >>>>>>>>> step.
> > >> > >>>>>>>>>>>>>>>>>>>>> Concepts
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> can
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> > >> > >>>>> below
> > >> > >>>>>>> the
> > >> > >>>>>>>>> desk
> > >> > >>>>>>>>>>>> with
> > >> > >>>>>>>>>>>>>>>>>>>>> you
> > >> > >>>>>>>>>>>>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
> > >> > >>> keep
> > >> > >>>>>>>> details
> > >> > >>>>>>>>> of
> > >> > >>>>>>>>>>>>>>>>>>> python
> > >> > >>>>>>>>>>>>>>>>>>>>>> names
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
> > >> > >>> DAGâŠ
> > >> > >>>>>> but
> > >> > >>>>>>>> user
> > >> > >>>>>>>>>>>> facing
> > >> > >>>>>>>>>>>>>>>>>>> it
> > >> > >>>>>>>>>>>>>>>>>>>>>> is a
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
> > >> > >>> Hatter <
> > >> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > >> > >>>>>>>>> ryan.hatter@astronomer.io
> > >> > >>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
> > >> > >>> swords... at
> > >> > >>>>>>> least
> > >> > >>>>>>>>> for
> > >> > >>>>>>>>>>>> now.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
> > >> > >>> meaning to
> > >> > >>>>>>>> Airflow
> > >> > >>>>>>>>>>>> users.
> > >> > >>>>>>>>>>>>>>>>>>>>>> Indeed,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> has
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
> > >> > >>>>>> mathematicians
> > >> > >>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>> software
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> > >> > >>>>>> matter.
> > >> > >>>>>>>> For
> > >> > >>>>>>>>>>>> someone
> > >> > >>>>>>>>>>>>>>>>>>>>> new
> > >> > >>>>>>>>>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
> > >> > >>>>> orchestration,
> > >> > >>>>>>> one
> > >> > >>>>>>>>> of
> > >> > >>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>> first
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
> > >> > >>> DAG?"
> > >> > >>>>>> The
> > >> > >>>>>>>>> answer
> > >> > >>>>>>>>>>> is
> > >> > >>>>>>>>>>>>>>>>>>>>> almost
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
> > >> > >>> graph.
> > >> > >>>>> You
> > >> > >>>>>>>> don't
> > >> > >>>>>>>>>>> need
> > >> > >>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>> worry
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
> > >> > >>>>> workflow."
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> > >> > >>>>> least
> > >> > >>>>>> a
> > >> > >>>>>>>>> couple
> > >> > >>>>>>>>>>>>>>>>>>>>> important
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
> > >> > >>> mentioned
> > >> > >>>>>>> above,
> > >> > >>>>>>>>> "DAG"
> > >> > >>>>>>>>>>>> is
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
> > >> > >>> confusing.
> > >> > >>>>>> We
> > >> > >>>>>>>> want
> > >> > >>>>>>>>>>>> Airflow
> > >> > >>>>>>>>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>> be
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
> > >> > >>> off the
> > >> > >>>>>> bat
> > >> > >>>>>>>>> creates
> > >> > >>>>>>>>>>>> an
> > >> > >>>>>>>>>>>>>>>>>>>>>> initial
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> > >> > >>>>>>> Concepts*:
> > >> > >>>>>>>>> The
> > >> > >>>>>>>>>>> DAG
> > >> > >>>>>>>>>>>>>>>>>>> is
> > >> > >>>>>>>>>>>>>>>>>>>>>> just
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
> > >> > >>> workflow. The
> > >> > >>>>>>>>> workflow
> > >> > >>>>>>>>>>>>>>>>>>> includes
> > >> > >>>>>>>>>>>>>>>>>>>>>> its
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> > >> > >>>>> other
> > >> > >>>>>>>>>>> parameters,
> > >> > >>>>>>>>>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>>>>> other
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
> > >> > >>> Airflow
> > >> > >>>>>>>> homepage
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > >> > >>>>>>>>>>> https://airflow.apache.org/>>.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
> > >> > >>> created by
> > >> > >>>>>> the
> > >> > >>>>>>>>>>> community
> > >> > >>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
> > >> > >>> workflows.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
> > >> > >>>>> Airflow?"
> > >> > >>>>>>> docs
> > >> > >>>>>>>>> page
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>
> https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > <
> > >> > >>>>>>>>>>>
> > >> > >>>>>
> > https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > >> > >>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
> > >> > >>> Airflow is
> > >> > >>>>>>>> without
> > >> > >>>>>>>>>>> using
> > >> > >>>>>>>>>>>>>>>>>>>>> "DAG."
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
> > >> > >>> term
> > >> > >>>>> is
> > >> > >>>>>>>>>>> introduced
> > >> > >>>>>>>>>>>>>>>>>>> out
> > >> > >>>>>>>>>>>>>>>>>>>>> of
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
> > >> > >>> explain
> > >> > >>>>> it:
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> > >> > >>>>> collection
> > >> > >>>>>>> of
> > >> > >>>>>>>>> tasks
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
> > >> > >>> in
> > >> > >>>>>> these
> > >> > >>>>>>>>>>>>>>>>>>> introductions
> > >> > >>>>>>>>>>>>>>>>>>>>> to
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> > >> > >>>>>> orchestrate
> > >> > >>>>>>>>> DAGs;
> > >> > >>>>>>>>>>> it
> > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> > >> > >>>>>> irrelevant
> > >> > >>>>>>>> to
> > >> > >>>>>>>>>>> almost
> > >> > >>>>>>>>>>>>>>>>>>>>> every
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
> > >> > >>> alias
> > >> > >>>>> for
> > >> > >>>>>>> the
> > >> > >>>>>>>>> term
> > >> > >>>>>>>>>>>> "DAG"
> > >> > >>>>>>>>>>>>>>>>>>>>> and
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
> > >> > >>> "DAG"
> > >> > >>>>>> with
> > >> > >>>>>>>>>>>> "workflow".
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
> > >> > >>>>> away.
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>
> ------------------------------------------------------------------
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > >> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > >> > >>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > >> > >>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
> > >> > >>>>>>>>>>> dev-help@airflow.apache.org>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >> > >>>>>>>>>>>>>>>>>>>>>
> > >> > >
> > >> > >
> > >> > >
> > ---------------------------------------------------------------------
> > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Constance Martineau <co...@astronomer.io.INVALID>.
Hey all,
I appreciate the discussion about how we position the concept of a DAG in
Airflow. While I agree with updating documentation to de-emphasize the
strict mathematical definition, I do not think we should change the public
interface to use "dag" instead of "DAG" everywhere. It adds
unnecessary churn and breaks consistency across the ecosystem.
Regarding the broader discussion on whether DAGs in Airflow should remain
acyclic: I acknowledge that some workflows, particularly in ML and AI,
require cyclical behaviour (example: iterative refinement until a condition
is met). However, I think it's important to recognize that even in those
cases, the workflow itself still has an overall start and end. It's not an
unbounded loop but rather a controlled execution process that happens to
have cycles *within* it.
That being said, I do see value in reinterpreting "DAG" to better align
with how Airflow is actually used, and love shifting away from "Directed
Acyclic Graph" in favour of something more intuitive. However "Data Airflow
Graph" makes it sound like we're defining Airflow itself rather than
describing the structure of workflows within it.
If we want a new meaning for DAG that resonates with our base while keeping
Airflow's identity clear, I am proposing three alternatives:
1. *Dynamic Airflow Graph*: Ties directly to Airflow while emphasizing
flexibility
2. *Distributed Automation Graph*: Highlights execution across systems
and automations without tying it strictly to data
3. *Directed Activity Graph*: Retains the useful meaning of "Directed"
while shifting away from strict acyclic constraints. (My personal favourite)
I think any shift in terminology should be driven by what best communicates
Airflow's role workflow automation, rather than just distancing ourselves
from the mathematical definition.
Curious to hear more thoughts!
On Tue, Feb 18, 2025 at 10:40âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> Sorry for the few typos ... My slight dyslexia did not help and autocorrect
> did not help either this time : dept -> debt, routed -> rooted and a few
> other small typos :) .
>
> On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > First of all - yes, agree with Lee and Ash, I think we do not need to
> > change the public interface, I'd also be -0 on that, but docs, internal
> > strings, yes we could.
> >
> > But .... After thinking a bit and looking at the discussion here, where
> > people have concerns, I have an alternative take. And I know this might
> be
> > controversial, and some people might find it confusing, but why don't we
> > come up with a new acronym for DAG, one that will be more airflow and
> less
> > "mathematics/algorithm". And I have, I think a good reason for it, and I
> > even have a proposal. Bear with me,
> >
> > DAG => *Data Airflow Graph*
> >
> > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now (with
> > exception of some niche products - you know which one I speak about -
> which
> > are unfortunate enough to use the "Dag" in their name - I will explain
> why
> > this is unfortunate, IMHO).
> >
> > We are lucky to have Airflow for "A" in the acronym and we can make it
> > even more "synonymous" -> Airflow <> DAG with (A) being Airflow.
> >
> > Now, why I think "Directed Acyclic Graph" is unfortunate.
> >
> > Not only because "Directed" and "Acyclic" are mostly not in the
> vocabulary
> > of most of our users - as Ryan, starting this thread explained yes, I
> agree
> > with it, but also I think there is a more important reason - I simply
> > believe in the future our DAGs might not (and likely will not) be
> "acyclic".
> >
> > I know for some it might be herezy and we have not discussed any of that
> > yet, but more and more I keep on hearing that Machine Learning Workflows
> of
> > the future are often somewhat cyclic. In our case there is one type of
> > repeat-in-cycle behaviour - we have "task retry on failure" - but this is
> > only on failure and it basically mens "idempotent" retry - we want to
> redo
> > something and get single result out of that and basically discard the
> > previous result. But machine learning/AI workflows are different. I had a
> > lot of conversations recently (I even co-organized "AI low-level
> > engineering and hacking" devroom to learn more about those) - and I keep
> on
> > hearing that "Repeat a sequence of things several times and refine" is
> > precisely what AI / ML workflows will need - especially with Agentic
> > workflows - often we have cooperating agents which **could** be part of
> the
> > same DAG, but currently in Airflow where our DAGs are "Acyclic" we can
> > only express "repeat several time and refine" as a single task that will
> do
> > such refining. If we would like to split such an "agent" doing their own
> > work in "cycles" and communicating with other "agents" doing the same -
> > represented as a multi-task-entity, we have no way to express it using
> our
> > current "acyclic" semantics.
> >
> > But - other than our implementation makes that assumption currently -
> > there is nothing fundamental with Airflow being able to run such
> "partially
> > cyclic workflows". We have all the pieces, we would have to change
> > dependency calculation to allow for cycles, update UI to handle cycles
> > representation - and we could in theory quite seamlessly, support
> "cyclic"
> > workflows.
> >
> > I know, I know, that's the first time we hear it and that's huge change -
> > of course, I do not want to start discussing any details of it, not
> before
> > Airflow 3.0 is out - but I think we should be open that at some point of
> > time the "Acyclic" property is going to go away, And my personal
> intuition
> > tells me that a year from now we will have acyclic workflows (assuming
> that
> > all the foundational work we do on Airflow 3 will really help in
> decreasing
> > all the technical dept, and will open our minds to new ideas).
> >
> > That's why I think - the sooner we get rid of "Directed Acyclic" - the
> > better. that will - even for us, maintainer free us from being routed in
> > the acyclic nature of Airflow.
> >
> > How about we change to "Data Airflow Graph" (or any other good acronym
> > which has (A)irflow instead of (A)cyclic). The name is just a proposal.
> >
> > Now.. I go hide somewhere as I believe that might cause a "flamewar" :),
> > unless on request of Ryan you already sheathed your swords in the
> > discussion (which seems everyone did).
> >
> > But honestly - I would really love what you think of "let's just break
> > with Acyclic NOW" and shape our future by using a different acronym.
> >
> > J.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> > ephraimanierobi@gmail.com> wrote:
> >
> >> When I started contributing to Airflow, I had to read up on DAG, and
> now I
> >> know what it means. I wonder if we are about to have users who use the
> >> term
> >> Dag without knowing that there's a DAG.
> >>
> >> Slightly concerned.
> >>
> >> -ephraim
> >>
> >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com> wrote:
> >>
> >> > For user-facing things (e.g., `from airflow import DAG`), Iâm more
> like
> >> > -0. But for documentation, docstring, internal things and etc., we
> >> probably
> >> > could still change most of them?
> >> >
> >> > Best,
> >> > Wei
> >> >
> >> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org>
> >> wrote:
> >> > >
> >> > > I also donât personally think itâs worth the pain (not to mention
> >> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
> >> > >
> >> > > -ash
> >> > >
> >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <
> daniel.imberman@gmail.com
> >> >
> >> > wrote:
> >> > >>
> >> > >> I think my biggest concern is a marketing one and not a technical
> >> one.
> >> > >>
> >> > >> As has been mentioned on the thread the terms airflow and dag are
> >> kind
> >> > of
> >> > >> synonymous and I certainly donât want to give the impression that
> we
> >> are
> >> > >> breaking more than we are breaking.
> >> > >>
> >> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> >> > >>
> >> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
> >> > >>
> >> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff rules
> are
> >> > best
> >> > >>> suited for user-facing things but not the airflow code base
> itself.
> >> If
> >> > what
> >> > >>> we mean is adding a rule to avoid users using "DAG" *after* we
> >> rename
> >> > it,
> >> > >>> it's definitely a +1000.
> >> > >>>
> >> > >>> I just created GitHub issues for this removing "DAG"
> >> > >>>
> >> > >>> https://github.com/apache/airflow/issues/46842
> >> > >>> https://github.com/apache/airflow/issues/46843
> >> > >>>
> >> > >>> One thing I'm not sure about is whether we want to get rid of
> `from
> >> > >>> airflow import DAG` as well. đ€
> >> > >>>
> >> > >>> Best,
> >> > >>> Wei
> >> > >>>
> >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> >> > >>>> Hard to say until it's looked at :)
> >> > >>>>
> >> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> >> aritrabasu1999@gmail.com
> >> > >
> >> > >>>> wrote:
> >> > >>>>
> >> > >>>>> I can take it up, it's mostly just a doc update right? Or are we
> >> > doing
> >> > >>> code
> >> > >>>>> files replacement too?
> >> > >>>>> --
> >> > >>>>> Regards,
> >> > >>>>> Aritra Basu
> >> > >>>>>
> >> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com>
> >> > wrote:
> >> > >>>>>
> >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any
> >> volunteers
> >> > >>> to
> >> > >>>>>> make it ? It's fun.
> >> > >>>>>>
> >> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <
> droiddev510e@gmail.com>
> >> > >>> wrote:
> >> > >>>>>>
> >> > >>>>>>> +1 for ruff rules :)
> >> > >>>>>>>
> >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the
> >> > >>> Airflow
> >> > >>>>>>> docs, in line with the new UI changes and to make the renaming
> >> > >>>>>>> consistent across user-facing pages.
> >> > >>>>>>>
> >> > >>>>>>> Regards,
> >> > >>>>>>> Omkar
> >> > >>>>>>>
> >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <
> jarek@potiuk.com>
> >> > >>> wrote:
> >> > >>>>>>>
> >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> >> > >>>>>>>>
> >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
> >> > >>> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>>> It seems that our current conclusion is to use "dag" or
> "Dag"
> >> > >>>>> instead
> >> > >>>>>>> of
> >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the
> >> > >>>>> codebase
> >> > >>>>>>> with
> >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be
> >> > >>> đ€),
> >> > >>>>>>> should
> >> > >>>>>>>> we
> >> > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
> >> > >>>>>>>>>
> >> > >>>>>>>>> Best,
> >> > >>>>>>>>> Wei
> >> > >>>>>>>>>
> >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
> >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> >> > >>>>>>>>>> Let me know if that's the direction we want to go.
> >> > >>>>>>>>>>
> >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> >> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
> >> > >>> synonymous
> >> > >>>>>> with
> >> > >>>>>>>>>>> workflows in Airflow through the way we talk and explain
> >> > >>> what
> >> > >>>>>>> Airflow
> >> > >>>>>>>>> is
> >> > >>>>>>>>>>> all about. Although folks would ask, I donât ever use the
> >> > >>>>>>>> mathematical
> >> > >>>>>>>>>>> definition of DAG.
> >> > >>>>>>>>>>> It will be challenging and possibly confusing for many
> >> > >>> users
> >> > >>>>>> making
> >> > >>>>>>>>> such a
> >> > >>>>>>>>>>> change - I would rather direct that energy to appending
> the
> >> > >>>>>> Oxford
> >> > >>>>>>>>>>> definition of "dag" to include a reference to workflows in
> >> > >>>>>> Airflow.
> >> > >>>>>>>> __
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> -- Rajesh
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> >> > >>>>>> <mailto:
> >> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> CAUTION: This email originated from outside of the
> >> > >>>>> organization.
> >> > >>>>>> Do
> >> > >>>>>>>> not
> >> > >>>>>>>>>>> click links or open attachments unless you can confirm the
> >> > >>>>> sender
> >> > >>>>>>> and
> >> > >>>>>>>>> know
> >> > >>>>>>>>>>> the content is safe.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> >> > >>>>> expéditeur
> >> > >>>>>>>>> externe.
> >> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
> >> > >>> si
> >> > >>>>> vous
> >> > >>>>>>> ne
> >> > >>>>>>>>> pouvez
> >> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
> >> > >>> pas
> >> > >>>>>>>> certain
> >> > >>>>>>>>> que
> >> > >>>>>>>>>>> le contenu ne présente aucun risque.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> From Guido's post:
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> "Around this time the renaming seems to have been
> renamed".
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> Naming is hard.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> J.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> >> > >>>>> droiddev510e@gmail.com
> >> > >>>>>>>>> <mailto:
> >> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an
> >> > >>> Airflow
> >> > >>>>>> term
> >> > >>>>>>>> now
> >> > >>>>>>>>>>>> rather
> >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not,
> >> > >>> I've
> >> > >>>>>>>> worked
> >> > >>>>>>>>> with
> >> > >>>>>>>>>>>> work
> >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now
> >> > >>> and are
> >> > >>>>>> pro
> >> > >>>>>>>>> devs
> >> > >>>>>>>>>>> but
> >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For
> >> > >>> them,
> >> > >>>>> dag
> >> > >>>>>> =
> >> > >>>>>>>>> Airflow.
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
> >> > >>> will be
> >> > >>>>> a
> >> > >>>>>>>> major
> >> > >>>>>>>>>>>> behavior
> >> > >>>>>>>>>>>> change for the developer community, so we'll need a solid
> >> > >>>>> plan
> >> > >>>>>> on
> >> > >>>>>>>>> how to
> >> > >>>>>>>>>>>> introduce it, when we do.
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the Great
> >> > >>>>>> Renaming
> >> > >>>>>>>> in
> >> > >>>>>>>>> core
> >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably
> >> > >>> get
> >> > >>>>>> some
> >> > >>>>>>>>>>> learnings
> >> > >>>>>>>>>>>> from there? Who knows!
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> Guido's blog (2009):
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>
> >> > >>>>>
> >> > >>>
> >> >
> >>
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> >> > >>>>>>>>>>> <
> >> > >>>>>>>>>
> >> > >>>>>>>
> >> > >>>>>
> >> > >>>
> >> >
> >>
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> >> > >>>>>>>>>
> >> > >>>>>>>>>>>> Follow-up discussion (2024):
> >> > >>>>>>>>>>>>
> >> > >>>>>>
> https://discuss.python.org/t/finishing-the-great-renaming/54082
> >> > >>>>>>> <
> >> > >>>>>>>>>>>
> >> > >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082
> >
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> Cheers,
> >> > >>>>>>>>>>>> Omkar
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> >> > >>> weilee.rx@gmail.com
> >> > >>>>>>>>> <mailto:
> >> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>>> I think we should probably just accept it as an airflow
> >> > >>>>> term.
> >> > >>>>>>> At
> >> > >>>>>>>>> least,
> >> > >>>>>>>>>>>>> thatâs how I understood it when I first used Airflow. I
> >> > >>>>> feel
> >> > >>>>>>>>> renaming
> >> > >>>>>>>>>>> it
> >> > >>>>>>>>>>>> at
> >> > >>>>>>>>>>>>> this stage would require considerable effort from
> >> > >>>>> maintainers
> >> > >>>>>>> and
> >> > >>>>>>>>>>>> existing
> >> > >>>>>>>>>>>>> users without providing equivalent benefits.
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>> Best,
> >> > >>>>>>>>>>>>> Wei
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> >> > >>>>>> kaxilnaik@gmail.com
> >> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
> >> > >>>>> this
> >> > >>>>>>>>> discussion
> >> > >>>>>>>>>>>> for
> >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
> >> > >>>>>>> disruption
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> >> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> >> > >>>>>>>>> constance@astronomer.io.inva>lid>
> >> > >>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
> >> > >>>>>> experience
> >> > >>>>>>>>> what a
> >> > >>>>>>>>>>>> dag
> >> > >>>>>>>>>>>>> is,
> >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes -
> >> > >>> stuff
> >> > >>>>>> like
> >> > >>>>>>>>> dags
> >> > >>>>>>>>>>>> being
> >> > >>>>>>>>>>>>> a
> >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
> >> > >>>>>> doesn't
> >> > >>>>>>>>> come up.
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> >> > >>> embrace
> >> > >>>>> the
> >> > >>>>>>> name
> >> > >>>>>>>>> and
> >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> >> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> >> > >>>>> vikram@astronomer.io.inva
> >> > >>>>>>>>> lid>
> >> > >>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> >> > >>>>> struggling
> >> > >>>>>>> with
> >> > >>>>>>>>> this
> >> > >>>>>>>>>>>>> when I
> >> > >>>>>>>>>>>>>>>> started working with Airflow.
> >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
> >> > >>>>>>>> established
> >> > >>>>>>>>>>>> concept
> >> > >>>>>>>>>>>>>>> now
> >> > >>>>>>>>>>>>>>>> regardless of the origin.
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
> >> > >>> best
> >> > >>>>>>>> expressed
> >> > >>>>>>>>> by
> >> > >>>>>>>>>>>>> Daniel
> >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
> >> > >>> than
> >> > >>>>>> the
> >> > >>>>>>>>> computer
> >> > >>>>>>>>>>>>>>> science
> >> > >>>>>>>>>>>>>>>> concept.
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> Best regards,
> >> > >>>>>>>>>>>>>>>> Vikram
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> >> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> >> > >>>>>> onikolas@amazon.com.inva
> >> > >>>>>>>>> lid
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
> >> > >>> right
> >> > >>>>>> Ryan,
> >> > >>>>>>>> DAG
> >> > >>>>>>>>>>> isn't
> >> > >>>>>>>>>>>>>>> great
> >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
> >> > >>> cause
> >> > >>>>> much
> >> > >>>>>>>> more
> >> > >>>>>>>>>>>> wreckage
> >> > >>>>>>>>>>>>>>>> than
> >> > >>>>>>>>>>>>>>>>> it solves.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
> >> > >>>>> defining
> >> > >>>>>>>> DAG.
> >> > >>>>>>>>> I
> >> > >>>>>>>>>>>> think
> >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
> >> > >>> for a
> >> > >>>>>> while
> >> > >>>>>>>> now
> >> > >>>>>>>>>>>> anyway,
> >> > >>>>>>>>>>>>>>> so
> >> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Cheers,
> >> > >>>>>>>>>>>>>>>>> Niko
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> ________________________________
> >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> >> > >>>>>>>>> ash@apache.org>>
> >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> >> > >>>>>> dev@airflow.apache.org
> >> > >>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
> >> > >>> term
> >> > >>>>>> "DAG"
> >> > >>>>>>>>> for end
> >> > >>>>>>>>>>>>>>> users
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
> >> > >>>>>>>>> organization.
> >> > >>>>>>>>>>> Do
> >> > >>>>>>>>>>>>> not
> >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can
> >> > >>> confirm
> >> > >>>>>> the
> >> > >>>>>>>>> sender
> >> > >>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>> know
> >> > >>>>>>>>>>>>>>>>> the content is safe.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> >> > >>> dâun
> >> > >>>>>>>>> expéditeur
> >> > >>>>>>>>>>>>>>> externe.
> >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> >> > >>>>>> jointe
> >> > >>>>>>> si
> >> > >>>>>>>>> vous
> >> > >>>>>>>>>>> ne
> >> > >>>>>>>>>>>>>>>> pouvez
> >> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> >> > >>> vous
> >> > >>>>>> nâĂȘtes
> >> > >>>>>>>> pas
> >> > >>>>>>>>>>>> certain
> >> > >>>>>>>>>>>>>>>> que
> >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
> >> > >>> term â
> >> > >>>>>>>> getting
> >> > >>>>>>>>> to
> >> > >>>>>>>>>>>>> re-use
> >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
> >> > >>>>> sides
> >> > >>>>>>> have
> >> > >>>>>>>>> good
> >> > >>>>>>>>>>>>>>> reasons
> >> > >>>>>>>>>>>>>>>>> presented.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> -a
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> >> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
> >> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> >> > >>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
> >> > >>> from,
> >> > >>>>>>> "well,
> >> > >>>>>>>>> no
> >> > >>>>>>>>>>> one
> >> > >>>>>>>>>>>>>>>>> actually
> >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> >> > >>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> >> > >>>>>>>>> jarek@potiuk.com
> >> > >>>>>>>>>>> <ma...@potiuk.com>>
> >> > >>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
> >> > >>> the
> >> > >>>>>>>>>>> tongue-twister,
> >> > >>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> >> > >>>>>>>>> jarek@potiuk.com
> >> > >>>>>>>>>>> <ma...@potiuk.com>>
> >> > >>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
> >> > >>> would
> >> > >>>>> very
> >> > >>>>>>>> much
> >> > >>>>>>>>> want
> >> > >>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>> be
> >> > >>>>>>>>>>>>>>>>>>> able
> >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> >> > >>> further.
> >> > >>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
> >> > >>> it's a
> >> > >>>>>>>>>>>> tongue-twister,
> >> > >>>>>>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>> I
> >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> >> > >>> pronounce
> >> > >>>>>> it
> >> > >>>>>>>>>>> properly.
> >> > >>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>> J.
> >> > >>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> >> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> >> > >>>>>>>> brent@astronomer.io.inva
> >> > >>>>>>>>>> lid>
> >> > >>>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
> >> > >>>>> starting
> >> > >>>>>> on
> >> > >>>>>>>>> AIP-38
> >> > >>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>> modernize
> >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
> >> > >>>>>>>>> descriptive of
> >> > >>>>>>>>>>>> what
> >> > >>>>>>>>>>>>>>>> one
> >> > >>>>>>>>>>>>>>>>>>> is
> >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
> >> > >>> is an
> >> > >>>>>>>>>>> implementation
> >> > >>>>>>>>>>>>>>>>> detail.
> >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
> >> > >>> point
> >> > >>>>>> "DAG"
> >> > >>>>>>>> has
> >> > >>>>>>>>>>> become
> >> > >>>>>>>>>>>>>>>> "dag"
> >> > >>>>>>>>>>>>>>>>>>> , a
> >> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> >> > >>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
> >> > >>>>>> discussion,
> >> > >>>>>>>>> Airflow
> >> > >>>>>>>>>>>>>>>> Summit
> >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
> >> > >>>>>> embrace
> >> > >>>>>>>>> "dag".
> >> > >>>>>>>>>>> A
> >> > >>>>>>>>>>>>>>> user
> >> > >>>>>>>>>>>>>>>>>>> just
> >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
> >> > >>>>> concept
> >> > >>>>>>>>> behind
> >> > >>>>>>>>>>> that
> >> > >>>>>>>>>>>>>>>>> word. I
> >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> >> > >>> refactoring so
> >> > >>>>>>> much
> >> > >>>>>>>>> code,
> >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
> >> > >>>>> questions,
> >> > >>>>>>>> etc.
> >> > >>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> >> > >>> Standish
> >> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> >> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
> >> > >>> of
> >> > >>>>>> pain
> >> > >>>>>>>> for
> >> > >>>>>>>>>>>>>>>>> questionable
> >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
> >> > >>> idea. I
> >> > >>>>>>> agree
> >> > >>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>> association
> >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
> >> > >>>>>>> invasive
> >> > >>>>>>>>>>>>>>> mitigations.
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
> >> > >>>>>> de-emphasize
> >> > >>>>>>>> the
> >> > >>>>>>>>>>> math
> >> > >>>>>>>>>>>>>>> nerd
> >> > >>>>>>>>>>>>>>>>>>>>> origins
> >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> >> > >>> website /
> >> > >>>>>> etc,
> >> > >>>>>>>>> *never
> >> > >>>>>>>>>>>>>>>> define*
> >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
> >> > >>>>>> graph.
> >> > >>>>>>>>> Always
> >> > >>>>>>>>>>>>>>> define
> >> > >>>>>>>>>>>>>>>>> it
> >> > >>>>>>>>>>>>>>>>>>>>> as a
> >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
> >> > >>> like a
> >> > >>>>>>>>> historical
> >> > >>>>>>>>>>>>>>>>> footnote,
> >> > >>>>>>>>>>>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
> >> > >>> hidden.
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
> >> > >>> docs
> >> > >>>>> in
> >> > >>>>>>>>> general
> >> > >>>>>>>>>>> e.g.
> >> > >>>>>>>>>>>>>>>>>>> writing
> >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> >> > >>> "DAGs"
> >> > >>>>>> etc.
> >> > >>>>>>>> The
> >> > >>>>>>>>>>>> upper
> >> > >>>>>>>>>>>>>>>> case
> >> > >>>>>>>>>>>>>>>>>>>>> part
> >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
> >> > >>> "dag" in
> >> > >>>>>>>>> airlfow is
> >> > >>>>>>>>>>>> just
> >> > >>>>>>>>>>>>>>>> an
> >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
> >> > >>> "DAGs"
> >> > >>>>> is
> >> > >>>>>>> not
> >> > >>>>>>>>> really
> >> > >>>>>>>>>>>>>>>>>>> unhelpful.
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
> >> > >>> is
> >> > >>>>> its
> >> > >>>>>>> own
> >> > >>>>>>>>>>> thing,
> >> > >>>>>>>>>>>> is
> >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
> >> > >>> nobody
> >> > >>>>>> knows
> >> > >>>>>>>>> about
> >> > >>>>>>>>>>>>>>>> anyway),
> >> > >>>>>>>>>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
> >> > >>> normal
> >> > >>>>>>> people
> >> > >>>>>>>>>>>>>>> understand.
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
> >> > >>> <
> >> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
> >> > >>> will
> >> > >>>>> be
> >> > >>>>>>>>> extremely
> >> > >>>>>>>>>>>>>>>>>>>>> difficult to
> >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
> >> > >>> will
> >> > >>>>>> make
> >> > >>>>>>> a
> >> > >>>>>>>>> lot of
> >> > >>>>>>>>>>>>>>>>>>> "google"
> >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
> >> > >>>>> finding
> >> > >>>>>>> the
> >> > >>>>>>>>> right
> >> > >>>>>>>>>>>>>>>>>>> answers.
> >> > >>>>>>>>>>>>>>>>>>>>>> This
> >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> >> > >>> besides the
> >> > >>>>>>>>> community
> >> > >>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>> ideas
> >> > >>>>>>>>>>>>>>>>>>>>> that
> >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
> >> > >>>>> examples,
> >> > >>>>>>>>> problems
> >> > >>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>>>> solutions
> >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> >> > >>> remember
> >> > >>>>>> that
> >> > >>>>>>>>> all the
> >> > >>>>>>>>>>>> AI
> >> > >>>>>>>>>>>>>>>>>>>>> trained
> >> > >>>>>>>>>>>>>>>>>>>>>> on
> >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> >> > >>> matching
> >> > >>>>>>> queries
> >> > >>>>>>>> of
> >> > >>>>>>>>>>>> people.
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
> >> > >>>>>> switch.
> >> > >>>>>>>>> And if
> >> > >>>>>>>>>>> we
> >> > >>>>>>>>>>>>>>> do
> >> > >>>>>>>>>>>>>>>> -
> >> > >>>>>>>>>>>>>>>>>>> I
> >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
> >> > >>> instead
> >> > >>>>> of
> >> > >>>>>>>>> `dag` if
> >> > >>>>>>>>>>>> not
> >> > >>>>>>>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>> above
> >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
> >> > >>> that it
> >> > >>>>>>> might
> >> > >>>>>>>>> cause
> >> > >>>>>>>>>>>> more
> >> > >>>>>>>>>>>>>>>>>>>>> problems
> >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
> >> > >>> think
> >> > >>>>>>> it's
> >> > >>>>>>>> a
> >> > >>>>>>>>> good
> >> > >>>>>>>>>>>>>>>> idea, I
> >> > >>>>>>>>>>>>>>>>>>>>> am
> >> > >>>>>>>>>>>>>>>>>>>>>> ok
> >> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>> J,
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> >> > >>> Bhakat
> >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> >> > >>> <mailto:
> >> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
> >> > >>> sense
> >> > >>>>> to
> >> > >>>>>>>>> someone
> >> > >>>>>>>>>>> new
> >> > >>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>> workflow
> >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
> >> > >>> nature
> >> > >>>>>> of
> >> > >>>>>>>>> being
> >> > >>>>>>>>>>>>>>> acyclic.
> >> > >>>>>>>>>>>>>>>>>>>>> Sure,
> >> > >>>>>>>>>>>>>>>>>>>>>>> as
> >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> >> > >>> workaround it.
> >> > >>>>>>>> Still,
> >> > >>>>>>>>> in
> >> > >>>>>>>>>>> my
> >> > >>>>>>>>>>>>>>>>>>>>> opinion,
> >> > >>>>>>>>>>>>>>>>>>>>>>> there
> >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
> >> > >>>>>>> workflow
> >> > >>>>>>>>>>>>>>>>>>> orchestration.
> >> > >>>>>>>>>>>>>>>>>>>>>> Most
> >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
> >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
> >> > >>>>> covered
> >> > >>>>>>>>> using an
> >> > >>>>>>>>>>>>>>>> acyclic
> >> > >>>>>>>>>>>>>>>>>>>>>> manner
> >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> >> > >>> idempotency. So
> >> > >>>>> I
> >> > >>>>>>>> would
> >> > >>>>>>>>> want
> >> > >>>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
> >> > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> >> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> >> > >>> bernd.stroehle@kosakya.de>>
> >> > >>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
> >> > >>>>> Airflow
> >> > >>>>>>> Fan;
> >> > >>>>>>>>> so
> >> > >>>>>>>>>>> many
> >> > >>>>>>>>>>>>>>> new
> >> > >>>>>>>>>>>>>>>>>>>>>> ideas.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
> >> > >>>>>> replaced
> >> > >>>>>>> by
> >> > >>>>>>>>> the
> >> > >>>>>>>>>>>> more
> >> > >>>>>>>>>>>>>>>>>>>>> general
> >> > >>>>>>>>>>>>>>>>>>>>>>>> Term
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> >> > >>>>>> (Petri)
> >> > >>>>>>>> Net
> >> > >>>>>>>>>>> (AFN)
> >> > >>>>>>>>>>>>>>>>>>> (maybe
> >> > >>>>>>>>>>>>>>>>>>>>>>>> without
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
> >> > >>>>> Graph
> >> > >>>>>>>>> Database.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> >> > >>> Graph
> >> > >>>>>> (AFG)
> >> > >>>>>>>>> might be
> >> > >>>>>>>>>>>>>>>>>>>>> assigned
> >> > >>>>>>>>>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>>> an
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
> >> > >>> of a
> >> > >>>>>>>> library.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
> >> > >>>>> structure
> >> > >>>>>>>> than
> >> > >>>>>>>>> a
> >> > >>>>>>>>>>>>>>>>>>>>> configuration
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> >> > >>>>>>>>>>> bernd.stroehle@gmail.com>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> >> > >>> <ikholopov@google.com.INVA
> >> > >>>>>>>>> <mailto:
> >> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> >> > >>>>>>>>> dev@airflow.apache.org
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
> >> > >>> term
> >> > >>>>>>> "DAG"
> >> > >>>>>>>>> for
> >> > >>>>>>>>>>> end
> >> > >>>>>>>>>>>>>>>>>>> users
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
> >> > >>>>>> suboptimal,
> >> > >>>>>>> it
> >> > >>>>>>>>> is
> >> > >>>>>>>>>>> part
> >> > >>>>>>>>>>>>>>> of
> >> > >>>>>>>>>>>>>>>>>>>>>> Airflow
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
> >> > >>> levels,
> >> > >>>>> that
> >> > >>>>>>> any
> >> > >>>>>>>>>>> attempt
> >> > >>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>> change
> >> > >>>>>>>>>>>>>>>>>>>>>>> it
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
> >> > >>> reduce it.
> >> > >>>>>> The
> >> > >>>>>>>>> only
> >> > >>>>>>>>>>>> thing
> >> > >>>>>>>>>>>>>>>>>>>>> that is
> >> > >>>>>>>>>>>>>>>>>>>>>>>> worse
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
> >> > >>> when
> >> > >>>>>>> there
> >> > >>>>>>>>> are
> >> > >>>>>>>>>>> two
> >> > >>>>>>>>>>>>>>> ways
> >> > >>>>>>>>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>>>> define
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
> >> > >>>>> tutorials
> >> > >>>>>>>> will
> >> > >>>>>>>>>>>> suddenly
> >> > >>>>>>>>>>>>>>>>>>>>> become
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
> >> > >>> as
> >> > >>>>>>> "DAG"s.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
> >> > >>> users
> >> > >>>>> away
> >> > >>>>>>>> with
> >> > >>>>>>>>> a
> >> > >>>>>>>>>>>> number
> >> > >>>>>>>>>>>>>>>>>>> of
> >> > >>>>>>>>>>>>>>>>>>>>>>>> breaking
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
> >> > >>>>>> breaking
> >> > >>>>>>>>> changes
> >> > >>>>>>>>>>>> for
> >> > >>>>>>>>>>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>>> most
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
> >> > >>> are
> >> > >>>>>>>> looking
> >> > >>>>>>>>> for.
> >> > >>>>>>>>>>>>>>>>>>>>> Attempting
> >> > >>>>>>>>>>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
> >> > >>>>> interpreted
> >> > >>>>>>> as
> >> > >>>>>>>> an
> >> > >>>>>>>>>>> even
> >> > >>>>>>>>>>>>>>>>>>>>> stronger
> >> > >>>>>>>>>>>>>>>>>>>>>>>> signal
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
> >> > >>> the
> >> > >>>>>> term
> >> > >>>>>>>> in
> >> > >>>>>>>>> the
> >> > >>>>>>>>>>>> long
> >> > >>>>>>>>>>>>>>>>>>>>> run. I
> >> > >>>>>>>>>>>>>>>>>>>>>>>> even
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
> >> > >>> it in
> >> > >>>>>> the
> >> > >>>>>>>> DAG
> >> > >>>>>>>>>>>>>>> definition
> >> > >>>>>>>>>>>>>>>>>>>>>>>> interface.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
> >> > >>> other
> >> > >>>>>> places
> >> > >>>>>>>> in
> >> > >>>>>>>>>>>> Airflow,
> >> > >>>>>>>>>>>>>>>>>>> of
> >> > >>>>>>>>>>>>>>>>>>>>>> which
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
> >> > >>> Harenslak
> >> > >>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
> >> > >>>>>>>> bas@astronomer.io.inva
> >> > >>>>>>>>>> lid
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
> >> > >>> have
> >> > >>>>>>> already
> >> > >>>>>>>>> faded
> >> > >>>>>>>>>>>> over
> >> > >>>>>>>>>>>>>>>>>>>>> time,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
> >> > >>>>>> create
> >> > >>>>>>>>> cyclic
> >> > >>>>>>>>>>>>>>>>>>> graphs,
> >> > >>>>>>>>>>>>>>>>>>>>>> e.g.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
> >> > >>>>> these
> >> > >>>>>>>>>>> properties
> >> > >>>>>>>>>>>>>>>>>>>>>> vanishing
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
> >> > >>>>>>> perspective I
> >> > >>>>>>>>>>> support
> >> > >>>>>>>>>>>>>>>>>>>>>> changing
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
> >> > >>>>>> naming:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
> >> > >>> name.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
> >> > >>> change, I
> >> > >>>>>>> suggest
> >> > >>>>>>>>>>>> defining
> >> > >>>>>>>>>>>>>>> a
> >> > >>>>>>>>>>>>>>>>>>>>>> clear
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
> >> > >>> the
> >> > >>>>>>>>> deprecation in
> >> > >>>>>>>>>>>>>>>>>>>>> Airflow
> >> > >>>>>>>>>>>>>>>>>>>>>> 3,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> >> > >>>>>>>> neil4real@gmail.com
> >> > >>>>>>>>>>> <ma...@gmail.com>>
> >> > >>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
> >> > >>>>>>>> especially
> >> > >>>>>>>>> when
> >> > >>>>>>>>>>>>>>>>>>> most
> >> > >>>>>>>>>>>>>>>>>>>>>> other
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
> >> > >>> wholeheartedly.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
> >> > >>>>> confusing
> >> > >>>>>>>>> about it
> >> > >>>>>>>>>>> at
> >> > >>>>>>>>>>>>>>>>>>>>> all,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
> >> > >>>>>>> confusing
> >> > >>>>>>>>> to
> >> > >>>>>>>>>>> most
> >> > >>>>>>>>>>>>>>>>>>>>> users
> >> > >>>>>>>>>>>>>>>>>>>>>>> who
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> >> > >>>>> Chung
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> >> > >>>>>>>>> tp@astronomer.io.inva
> >> > >>>>>>>>>>>> lid
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
> >> > >>> the
> >> > >>>>> term
> >> > >>>>>>>> DAG.
> >> > >>>>>>>>> The
> >> > >>>>>>>>>>>> only
> >> > >>>>>>>>>>>>>>>>>>>>>>> problem
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
> >> > >>>>>>> myselfâto
> >> > >>>>>>>>> stop
> >> > >>>>>>>>>>>> using
> >> > >>>>>>>>>>>>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
> >> > >>> mentioned
> >> > >>>>>>>>> âworkflowâ is
> >> > >>>>>>>>>>>> not
> >> > >>>>>>>>>>>>>>>>>>>>> very
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> >> > >>>>>>> proposed,
> >> > >>>>>>>>> but
> >> > >>>>>>>>>>>> itâs a
> >> > >>>>>>>>>>>>>>>>>>>>> bit
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
> >> > >>> places
> >> > >>>>> and
> >> > >>>>>>> kept
> >> > >>>>>>>>>>> finding
> >> > >>>>>>>>>>>>>>>>>>>>> myself
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
> >> > >>> wanting to
> >> > >>>>>>>> clarify
> >> > >>>>>>>>> âDAG
> >> > >>>>>>>>>>>>>>>>>>>>>>> definitionâ
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
> >> > >>> Scheffler
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> >> > >>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
> >> > >>> exactly
> >> > >>>>> same
> >> > >>>>>>>>>>>> observation,
> >> > >>>>>>>>>>>>>>>>>>>>> had a
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
> >> > >>> always
> >> > >>>>> an
> >> > >>>>>>>>>>> introduction
> >> > >>>>>>>>>>>>>>>>>>> if
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
> >> > >>> makes
> >> > >>>>>>> sense.
> >> > >>>>>>>>>>>>>>>>>>> Especially
> >> > >>>>>>>>>>>>>>>>>>>>>> when
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> also
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> >> > >>>>>>> reasonable
> >> > >>>>>>>>> step.
> >> > >>>>>>>>>>>>>>>>>>>>> Concepts
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> can
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> >> > >>>>> below
> >> > >>>>>>> the
> >> > >>>>>>>>> desk
> >> > >>>>>>>>>>>> with
> >> > >>>>>>>>>>>>>>>>>>>>> you
> >> > >>>>>>>>>>>>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
> >> > >>> keep
> >> > >>>>>>>> details
> >> > >>>>>>>>> of
> >> > >>>>>>>>>>>>>>>>>>> python
> >> > >>>>>>>>>>>>>>>>>>>>>> names
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
> >> > >>> DAGâŠ
> >> > >>>>>> but
> >> > >>>>>>>> user
> >> > >>>>>>>>>>>> facing
> >> > >>>>>>>>>>>>>>>>>>> it
> >> > >>>>>>>>>>>>>>>>>>>>>> is a
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
> >> > >>> Hatter <
> >> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> >> > >>>>>>>>> ryan.hatter@astronomer.io
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
> >> > >>> swords... at
> >> > >>>>>>> least
> >> > >>>>>>>>> for
> >> > >>>>>>>>>>>> now.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
> >> > >>> meaning to
> >> > >>>>>>>> Airflow
> >> > >>>>>>>>>>>> users.
> >> > >>>>>>>>>>>>>>>>>>>>>> Indeed,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> has
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
> >> > >>>>>> mathematicians
> >> > >>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>> software
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> >> > >>>>>> matter.
> >> > >>>>>>>> For
> >> > >>>>>>>>>>>> someone
> >> > >>>>>>>>>>>>>>>>>>>>> new
> >> > >>>>>>>>>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
> >> > >>>>> orchestration,
> >> > >>>>>>> one
> >> > >>>>>>>>> of
> >> > >>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>> first
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
> >> > >>> DAG?"
> >> > >>>>>> The
> >> > >>>>>>>>> answer
> >> > >>>>>>>>>>> is
> >> > >>>>>>>>>>>>>>>>>>>>> almost
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
> >> > >>> graph.
> >> > >>>>> You
> >> > >>>>>>>> don't
> >> > >>>>>>>>>>> need
> >> > >>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>> worry
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
> >> > >>>>> workflow."
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> >> > >>>>> least
> >> > >>>>>> a
> >> > >>>>>>>>> couple
> >> > >>>>>>>>>>>>>>>>>>>>> important
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
> >> > >>> mentioned
> >> > >>>>>>> above,
> >> > >>>>>>>>> "DAG"
> >> > >>>>>>>>>>>> is
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
> >> > >>> confusing.
> >> > >>>>>> We
> >> > >>>>>>>> want
> >> > >>>>>>>>>>>> Airflow
> >> > >>>>>>>>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>> be
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
> >> > >>> off the
> >> > >>>>>> bat
> >> > >>>>>>>>> creates
> >> > >>>>>>>>>>>> an
> >> > >>>>>>>>>>>>>>>>>>>>>> initial
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> >> > >>>>>>> Concepts*:
> >> > >>>>>>>>> The
> >> > >>>>>>>>>>> DAG
> >> > >>>>>>>>>>>>>>>>>>> is
> >> > >>>>>>>>>>>>>>>>>>>>>> just
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
> >> > >>> workflow. The
> >> > >>>>>>>>> workflow
> >> > >>>>>>>>>>>>>>>>>>> includes
> >> > >>>>>>>>>>>>>>>>>>>>>> its
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> >> > >>>>> other
> >> > >>>>>>>>>>> parameters,
> >> > >>>>>>>>>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>>>>> other
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
> >> > >>> Airflow
> >> > >>>>>>>> homepage
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> >> > >>>>>>>>>>> https://airflow.apache.org/>>.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
> >> > >>> created by
> >> > >>>>>> the
> >> > >>>>>>>>>>> community
> >> > >>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
> >> > >>> workflows.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
> >> > >>>>> Airflow?"
> >> > >>>>>>> docs
> >> > >>>>>>>>> page
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html
> <
> >> > >>>>>>>>>>>
> >> > >>>>>
> https://airflow.apache.org/docs/apache-airflow/stable/index.html
> >> > >>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
> >> > >>> Airflow is
> >> > >>>>>>>> without
> >> > >>>>>>>>>>> using
> >> > >>>>>>>>>>>>>>>>>>>>> "DAG."
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
> >> > >>> term
> >> > >>>>> is
> >> > >>>>>>>>>>> introduced
> >> > >>>>>>>>>>>>>>>>>>> out
> >> > >>>>>>>>>>>>>>>>>>>>> of
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
> >> > >>> explain
> >> > >>>>> it:
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> >> > >>>>> collection
> >> > >>>>>>> of
> >> > >>>>>>>>> tasks
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
> >> > >>> in
> >> > >>>>>> these
> >> > >>>>>>>>>>>>>>>>>>> introductions
> >> > >>>>>>>>>>>>>>>>>>>>> to
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> >> > >>>>>> orchestrate
> >> > >>>>>>>>> DAGs;
> >> > >>>>>>>>>>> it
> >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates
> >> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> >> > >>>>>> irrelevant
> >> > >>>>>>>> to
> >> > >>>>>>>>>>> almost
> >> > >>>>>>>>>>>>>>>>>>>>> every
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
> >> > >>> alias
> >> > >>>>> for
> >> > >>>>>>> the
> >> > >>>>>>>>> term
> >> > >>>>>>>>>>>> "DAG"
> >> > >>>>>>>>>>>>>>>>>>>>> and
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
> >> > >>> "DAG"
> >> > >>>>>> with
> >> > >>>>>>>>>>>> "workflow".
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
> >> > >>>>> away.
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>> ------------------------------------------------------------------
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> >> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> >> > >>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> >> > >>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
> >> > >>>>>>>>>>> dev-help@airflow.apache.org>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>>>>>>
> >> > >
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> >> > >
> >> >
> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > For additional commands, e-mail: dev-help@airflow.apache.org
> >> >
> >> >
> >>
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Sorry for the few typos ... My slight dyslexia did not help and autocorrect
did not help either this time : dept -> debt, routed -> rooted and a few
other small typos :) .
On Tue, Feb 18, 2025 at 4:28âŻPM Jarek Potiuk <ja...@potiuk.com> wrote:
> First of all - yes, agree with Lee and Ash, I think we do not need to
> change the public interface, I'd also be -0 on that, but docs, internal
> strings, yes we could.
>
> But .... After thinking a bit and looking at the discussion here, where
> people have concerns, I have an alternative take. And I know this might be
> controversial, and some people might find it confusing, but why don't we
> come up with a new acronym for DAG, one that will be more airflow and less
> "mathematics/algorithm". And I have, I think a good reason for it, and I
> even have a proposal. Bear with me,
>
> DAG => *Data Airflow Graph*
>
> As many of us mentioned, DAG and Airflow are almost 100% 1-1 now (with
> exception of some niche products - you know which one I speak about - which
> are unfortunate enough to use the "Dag" in their name - I will explain why
> this is unfortunate, IMHO).
>
> We are lucky to have Airflow for "A" in the acronym and we can make it
> even more "synonymous" -> Airflow <> DAG with (A) being Airflow.
>
> Now, why I think "Directed Acyclic Graph" is unfortunate.
>
> Not only because "Directed" and "Acyclic" are mostly not in the vocabulary
> of most of our users - as Ryan, starting this thread explained yes, I agree
> with it, but also I think there is a more important reason - I simply
> believe in the future our DAGs might not (and likely will not) be "acyclic".
>
> I know for some it might be herezy and we have not discussed any of that
> yet, but more and more I keep on hearing that Machine Learning Workflows of
> the future are often somewhat cyclic. In our case there is one type of
> repeat-in-cycle behaviour - we have "task retry on failure" - but this is
> only on failure and it basically mens "idempotent" retry - we want to redo
> something and get single result out of that and basically discard the
> previous result. But machine learning/AI workflows are different. I had a
> lot of conversations recently (I even co-organized "AI low-level
> engineering and hacking" devroom to learn more about those) - and I keep on
> hearing that "Repeat a sequence of things several times and refine" is
> precisely what AI / ML workflows will need - especially with Agentic
> workflows - often we have cooperating agents which **could** be part of the
> same DAG, but currently in Airflow where our DAGs are "Acyclic" we can
> only express "repeat several time and refine" as a single task that will do
> such refining. If we would like to split such an "agent" doing their own
> work in "cycles" and communicating with other "agents" doing the same -
> represented as a multi-task-entity, we have no way to express it using our
> current "acyclic" semantics.
>
> But - other than our implementation makes that assumption currently -
> there is nothing fundamental with Airflow being able to run such "partially
> cyclic workflows". We have all the pieces, we would have to change
> dependency calculation to allow for cycles, update UI to handle cycles
> representation - and we could in theory quite seamlessly, support "cyclic"
> workflows.
>
> I know, I know, that's the first time we hear it and that's huge change -
> of course, I do not want to start discussing any details of it, not before
> Airflow 3.0 is out - but I think we should be open that at some point of
> time the "Acyclic" property is going to go away, And my personal intuition
> tells me that a year from now we will have acyclic workflows (assuming that
> all the foundational work we do on Airflow 3 will really help in decreasing
> all the technical dept, and will open our minds to new ideas).
>
> That's why I think - the sooner we get rid of "Directed Acyclic" - the
> better. that will - even for us, maintainer free us from being routed in
> the acyclic nature of Airflow.
>
> How about we change to "Data Airflow Graph" (or any other good acronym
> which has (A)irflow instead of (A)cyclic). The name is just a proposal.
>
> Now.. I go hide somewhere as I believe that might cause a "flamewar" :),
> unless on request of Ryan you already sheathed your swords in the
> discussion (which seems everyone did).
>
> But honestly - I would really love what you think of "let's just break
> with Acyclic NOW" and shape our future by using a different acronym.
>
> J.
>
>
>
>
>
>
>
>
>
>
> On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <
> ephraimanierobi@gmail.com> wrote:
>
>> When I started contributing to Airflow, I had to read up on DAG, and now I
>> know what it means. I wonder if we are about to have users who use the
>> term
>> Dag without knowing that there's a DAG.
>>
>> Slightly concerned.
>>
>> -ephraim
>>
>> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com> wrote:
>>
>> > For user-facing things (e.g., `from airflow import DAG`), Iâm more like
>> > -0. But for documentation, docstring, internal things and etc., we
>> probably
>> > could still change most of them?
>> >
>> > Best,
>> > Wei
>> >
>> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org>
>> wrote:
>> > >
>> > > I also donât personally think itâs worth the pain (not to mention
>> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
>> > >
>> > > -ash
>> > >
>> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <daniel.imberman@gmail.com
>> >
>> > wrote:
>> > >>
>> > >> I think my biggest concern is a marketing one and not a technical
>> one.
>> > >>
>> > >> As has been mentioned on the thread the terms airflow and dag are
>> kind
>> > of
>> > >> synonymous and I certainly donât want to give the impression that we
>> are
>> > >> breaking more than we are breaking.
>> > >>
>> > >> I wouldnât die on this hill, but Iâm slightly concerned.
>> > >>
>> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
>> > >>
>> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff rules are
>> > best
>> > >>> suited for user-facing things but not the airflow code base itself.
>> If
>> > what
>> > >>> we mean is adding a rule to avoid users using "DAG" *after* we
>> rename
>> > it,
>> > >>> it's definitely a +1000.
>> > >>>
>> > >>> I just created GitHub issues for this removing "DAG"
>> > >>>
>> > >>> https://github.com/apache/airflow/issues/46842
>> > >>> https://github.com/apache/airflow/issues/46843
>> > >>>
>> > >>> One thing I'm not sure about is whether we want to get rid of `from
>> > >>> airflow import DAG` as well. đ€
>> > >>>
>> > >>> Best,
>> > >>> Wei
>> > >>>
>> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
>> > >>>> Hard to say until it's looked at :)
>> > >>>>
>> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
>> aritrabasu1999@gmail.com
>> > >
>> > >>>> wrote:
>> > >>>>
>> > >>>>> I can take it up, it's mostly just a doc update right? Or are we
>> > doing
>> > >>> code
>> > >>>>> files replacement too?
>> > >>>>> --
>> > >>>>> Regards,
>> > >>>>> Aritra Basu
>> > >>>>>
>> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com>
>> > wrote:
>> > >>>>>
>> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any
>> volunteers
>> > >>> to
>> > >>>>>> make it ? It's fun.
>> > >>>>>>
>> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com>
>> > >>> wrote:
>> > >>>>>>
>> > >>>>>>> +1 for ruff rules :)
>> > >>>>>>>
>> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the
>> > >>> Airflow
>> > >>>>>>> docs, in line with the new UI changes and to make the renaming
>> > >>>>>>> consistent across user-facing pages.
>> > >>>>>>>
>> > >>>>>>> Regards,
>> > >>>>>>> Omkar
>> > >>>>>>>
>> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com>
>> > >>> wrote:
>> > >>>>>>>
>> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
>> > >>>>>>>>
>> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
>> > >>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag"
>> > >>>>> instead
>> > >>>>>>> of
>> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the
>> > >>>>> codebase
>> > >>>>>>> with
>> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be
>> > >>> đ€),
>> > >>>>>>> should
>> > >>>>>>>> we
>> > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
>> > >>>>>>>>>
>> > >>>>>>>>> Best,
>> > >>>>>>>>> Wei
>> > >>>>>>>>>
>> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
>> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
>> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
>> > >>>>>>>>>> Let me know if that's the direction we want to go.
>> > >>>>>>>>>>
>> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
>> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
>> > >>> synonymous
>> > >>>>>> with
>> > >>>>>>>>>>> workflows in Airflow through the way we talk and explain
>> > >>> what
>> > >>>>>>> Airflow
>> > >>>>>>>>> is
>> > >>>>>>>>>>> all about. Although folks would ask, I donât ever use the
>> > >>>>>>>> mathematical
>> > >>>>>>>>>>> definition of DAG.
>> > >>>>>>>>>>> It will be challenging and possibly confusing for many
>> > >>> users
>> > >>>>>> making
>> > >>>>>>>>> such a
>> > >>>>>>>>>>> change - I would rather direct that energy to appending the
>> > >>>>>> Oxford
>> > >>>>>>>>>>> definition of "dag" to include a reference to workflows in
>> > >>>>>> Airflow.
>> > >>>>>>>> __
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> -- Rajesh
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
>> > >>>>>> <mailto:
>> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> CAUTION: This email originated from outside of the
>> > >>>>> organization.
>> > >>>>>> Do
>> > >>>>>>>> not
>> > >>>>>>>>>>> click links or open attachments unless you can confirm the
>> > >>>>> sender
>> > >>>>>>> and
>> > >>>>>>>>> know
>> > >>>>>>>>>>> the content is safe.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
>> > >>>>> expéditeur
>> > >>>>>>>>> externe.
>> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
>> > >>> si
>> > >>>>> vous
>> > >>>>>>> ne
>> > >>>>>>>>> pouvez
>> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
>> > >>> pas
>> > >>>>>>>> certain
>> > >>>>>>>>> que
>> > >>>>>>>>>>> le contenu ne présente aucun risque.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> From Guido's post:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> "Around this time the renaming seems to have been renamed".
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Naming is hard.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> J.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
>> > >>>>> droiddev510e@gmail.com
>> > >>>>>>>>> <mailto:
>> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an
>> > >>> Airflow
>> > >>>>>> term
>> > >>>>>>>> now
>> > >>>>>>>>>>>> rather
>> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not,
>> > >>> I've
>> > >>>>>>>> worked
>> > >>>>>>>>> with
>> > >>>>>>>>>>>> work
>> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now
>> > >>> and are
>> > >>>>>> pro
>> > >>>>>>>>> devs
>> > >>>>>>>>>>> but
>> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For
>> > >>> them,
>> > >>>>> dag
>> > >>>>>> =
>> > >>>>>>>>> Airflow.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
>> > >>> will be
>> > >>>>> a
>> > >>>>>>>> major
>> > >>>>>>>>>>>> behavior
>> > >>>>>>>>>>>> change for the developer community, so we'll need a solid
>> > >>>>> plan
>> > >>>>>> on
>> > >>>>>>>>> how to
>> > >>>>>>>>>>>> introduce it, when we do.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> While we discuss this, I'd like to share about the Great
>> > >>>>>> Renaming
>> > >>>>>>>> in
>> > >>>>>>>>> core
>> > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably
>> > >>> get
>> > >>>>>> some
>> > >>>>>>>>>>> learnings
>> > >>>>>>>>>>>> from there? Who knows!
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Guido's blog (2009):
>> > >>>>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>
>> > >>>>>
>> > >>>
>> >
>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
>> > >>>>>>>>>>> <
>> > >>>>>>>>>
>> > >>>>>>>
>> > >>>>>
>> > >>>
>> >
>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
>> > >>>>>>>>>
>> > >>>>>>>>>>>> Follow-up discussion (2024):
>> > >>>>>>>>>>>>
>> > >>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082
>> > >>>>>>> <
>> > >>>>>>>>>>>
>> > >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Cheers,
>> > >>>>>>>>>>>> Omkar
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
>> > >>> weilee.rx@gmail.com
>> > >>>>>>>>> <mailto:
>> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> I think we should probably just accept it as an airflow
>> > >>>>> term.
>> > >>>>>>> At
>> > >>>>>>>>> least,
>> > >>>>>>>>>>>>> thatâs how I understood it when I first used Airflow. I
>> > >>>>> feel
>> > >>>>>>>>> renaming
>> > >>>>>>>>>>> it
>> > >>>>>>>>>>>> at
>> > >>>>>>>>>>>>> this stage would require considerable effort from
>> > >>>>> maintainers
>> > >>>>>>> and
>> > >>>>>>>>>>>> existing
>> > >>>>>>>>>>>>> users without providing equivalent benefits.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>> Wei
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
>> > >>>>>> kaxilnaik@gmail.com
>> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
>> > >>>>> this
>> > >>>>>>>>> discussion
>> > >>>>>>>>>>>> for
>> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
>> > >>>>>>> disruption
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
>> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
>> > >>>>>>>>> constance@astronomer.io.inva>lid>
>> > >>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
>> > >>>>>> experience
>> > >>>>>>>>> what a
>> > >>>>>>>>>>>> dag
>> > >>>>>>>>>>>>> is,
>> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes -
>> > >>> stuff
>> > >>>>>> like
>> > >>>>>>>>> dags
>> > >>>>>>>>>>>> being
>> > >>>>>>>>>>>>> a
>> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
>> > >>>>>> doesn't
>> > >>>>>>>>> come up.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
>> > >>> embrace
>> > >>>>> the
>> > >>>>>>> name
>> > >>>>>>>>> and
>> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
>> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
>> > >>>>> vikram@astronomer.io.inva
>> > >>>>>>>>> lid>
>> > >>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
>> > >>>>> struggling
>> > >>>>>>> with
>> > >>>>>>>>> this
>> > >>>>>>>>>>>>> when I
>> > >>>>>>>>>>>>>>>> started working with Airflow.
>> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
>> > >>>>>>>> established
>> > >>>>>>>>>>>> concept
>> > >>>>>>>>>>>>>>> now
>> > >>>>>>>>>>>>>>>> regardless of the origin.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
>> > >>> best
>> > >>>>>>>> expressed
>> > >>>>>>>>> by
>> > >>>>>>>>>>>>> Daniel
>> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
>> > >>> than
>> > >>>>>> the
>> > >>>>>>>>> computer
>> > >>>>>>>>>>>>>>> science
>> > >>>>>>>>>>>>>>>> concept.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> Best regards,
>> > >>>>>>>>>>>>>>>> Vikram
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
>> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
>> > >>>>>> onikolas@amazon.com.inva
>> > >>>>>>>>> lid
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
>> > >>> right
>> > >>>>>> Ryan,
>> > >>>>>>>> DAG
>> > >>>>>>>>>>> isn't
>> > >>>>>>>>>>>>>>> great
>> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
>> > >>> cause
>> > >>>>> much
>> > >>>>>>>> more
>> > >>>>>>>>>>>> wreckage
>> > >>>>>>>>>>>>>>>> than
>> > >>>>>>>>>>>>>>>>> it solves.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
>> > >>>>> defining
>> > >>>>>>>> DAG.
>> > >>>>>>>>> I
>> > >>>>>>>>>>>> think
>> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
>> > >>> for a
>> > >>>>>> while
>> > >>>>>>>> now
>> > >>>>>>>>>>>> anyway,
>> > >>>>>>>>>>>>>>> so
>> > >>>>>>>>>>>>>>>>> that feels like a natural step.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> Cheers,
>> > >>>>>>>>>>>>>>>>> Niko
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> ________________________________
>> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
>> > >>>>>>>>> ash@apache.org>>
>> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
>> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
>> > >>>>>> dev@airflow.apache.org
>> > >>>>>>>>
>> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
>> > >>> term
>> > >>>>>> "DAG"
>> > >>>>>>>>> for end
>> > >>>>>>>>>>>>>>> users
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
>> > >>>>>>>>> organization.
>> > >>>>>>>>>>> Do
>> > >>>>>>>>>>>>> not
>> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can
>> > >>> confirm
>> > >>>>>> the
>> > >>>>>>>>> sender
>> > >>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>> know
>> > >>>>>>>>>>>>>>>>> the content is safe.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
>> > >>> dâun
>> > >>>>>>>>> expéditeur
>> > >>>>>>>>>>>>>>> externe.
>> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
>> > >>>>>> jointe
>> > >>>>>>> si
>> > >>>>>>>>> vous
>> > >>>>>>>>>>> ne
>> > >>>>>>>>>>>>>>>> pouvez
>> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
>> > >>> vous
>> > >>>>>> nâĂȘtes
>> > >>>>>>>> pas
>> > >>>>>>>>>>>> certain
>> > >>>>>>>>>>>>>>>> que
>> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
>> > >>> term â
>> > >>>>>>>> getting
>> > >>>>>>>>> to
>> > >>>>>>>>>>>>> re-use
>> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
>> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
>> > >>>>> sides
>> > >>>>>>> have
>> > >>>>>>>>> good
>> > >>>>>>>>>>>>>>> reasons
>> > >>>>>>>>>>>>>>>>> presented.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> -a
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
>> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
>> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
>> > >>> from,
>> > >>>>>>> "well,
>> > >>>>>>>>> no
>> > >>>>>>>>>>> one
>> > >>>>>>>>>>>>>>>>> actually
>> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
>> > >>>>>>>>> jarek@potiuk.com
>> > >>>>>>>>>>> <ma...@potiuk.com>>
>> > >>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
>> > >>> the
>> > >>>>>>>>>>> tongue-twister,
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
>> > >>>>>>>>> jarek@potiuk.com
>> > >>>>>>>>>>> <ma...@potiuk.com>>
>> > >>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
>> > >>> would
>> > >>>>> very
>> > >>>>>>>> much
>> > >>>>>>>>> want
>> > >>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>> be
>> > >>>>>>>>>>>>>>>>>>> able
>> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
>> > >>> further.
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
>> > >>> it's a
>> > >>>>>>>>>>>> tongue-twister,
>> > >>>>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>> I
>> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
>> > >>> pronounce
>> > >>>>>> it
>> > >>>>>>>>>>> properly.
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> J.
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
>> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
>> > >>>>>>>> brent@astronomer.io.inva
>> > >>>>>>>>>> lid>
>> > >>>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
>> > >>>>> starting
>> > >>>>>> on
>> > >>>>>>>>> AIP-38
>> > >>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>> modernize
>> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
>> > >>>>>>>>> descriptive of
>> > >>>>>>>>>>>> what
>> > >>>>>>>>>>>>>>>> one
>> > >>>>>>>>>>>>>>>>>>> is
>> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
>> > >>> is an
>> > >>>>>>>>>>> implementation
>> > >>>>>>>>>>>>>>>>> detail.
>> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
>> > >>> point
>> > >>>>>> "DAG"
>> > >>>>>>>> has
>> > >>>>>>>>>>> become
>> > >>>>>>>>>>>>>>>> "dag"
>> > >>>>>>>>>>>>>>>>>>> , a
>> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
>> > >>>>>> discussion,
>> > >>>>>>>>> Airflow
>> > >>>>>>>>>>>>>>>> Summit
>> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
>> > >>>>>> embrace
>> > >>>>>>>>> "dag".
>> > >>>>>>>>>>> A
>> > >>>>>>>>>>>>>>> user
>> > >>>>>>>>>>>>>>>>>>> just
>> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
>> > >>>>> concept
>> > >>>>>>>>> behind
>> > >>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>> word. I
>> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
>> > >>> refactoring so
>> > >>>>>>> much
>> > >>>>>>>>> code,
>> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
>> > >>>>> questions,
>> > >>>>>>>> etc.
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
>> > >>> Standish
>> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
>> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
>> > >>> of
>> > >>>>>> pain
>> > >>>>>>>> for
>> > >>>>>>>>>>>>>>>>> questionable
>> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
>> > >>> idea. I
>> > >>>>>>> agree
>> > >>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>> association
>> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
>> > >>>>>>> invasive
>> > >>>>>>>>>>>>>>> mitigations.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
>> > >>>>>> de-emphasize
>> > >>>>>>>> the
>> > >>>>>>>>>>> math
>> > >>>>>>>>>>>>>>> nerd
>> > >>>>>>>>>>>>>>>>>>>>> origins
>> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
>> > >>> website /
>> > >>>>>> etc,
>> > >>>>>>>>> *never
>> > >>>>>>>>>>>>>>>> define*
>> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
>> > >>>>>> graph.
>> > >>>>>>>>> Always
>> > >>>>>>>>>>>>>>> define
>> > >>>>>>>>>>>>>>>>> it
>> > >>>>>>>>>>>>>>>>>>>>> as a
>> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
>> > >>> like a
>> > >>>>>>>>> historical
>> > >>>>>>>>>>>>>>>>> footnote,
>> > >>>>>>>>>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
>> > >>> hidden.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
>> > >>> docs
>> > >>>>> in
>> > >>>>>>>>> general
>> > >>>>>>>>>>> e.g.
>> > >>>>>>>>>>>>>>>>>>> writing
>> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
>> > >>> "DAGs"
>> > >>>>>> etc.
>> > >>>>>>>> The
>> > >>>>>>>>>>>> upper
>> > >>>>>>>>>>>>>>>> case
>> > >>>>>>>>>>>>>>>>>>>>> part
>> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
>> > >>> "dag" in
>> > >>>>>>>>> airlfow is
>> > >>>>>>>>>>>> just
>> > >>>>>>>>>>>>>>>> an
>> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
>> > >>> "DAGs"
>> > >>>>> is
>> > >>>>>>> not
>> > >>>>>>>>> really
>> > >>>>>>>>>>>>>>>>>>> unhelpful.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
>> > >>> is
>> > >>>>> its
>> > >>>>>>> own
>> > >>>>>>>>>>> thing,
>> > >>>>>>>>>>>> is
>> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
>> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
>> > >>> nobody
>> > >>>>>> knows
>> > >>>>>>>>> about
>> > >>>>>>>>>>>>>>>> anyway),
>> > >>>>>>>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
>> > >>> normal
>> > >>>>>>> people
>> > >>>>>>>>>>>>>>> understand.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
>> > >>> <
>> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
>> > >>> will
>> > >>>>> be
>> > >>>>>>>>> extremely
>> > >>>>>>>>>>>>>>>>>>>>> difficult to
>> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
>> > >>> will
>> > >>>>>> make
>> > >>>>>>> a
>> > >>>>>>>>> lot of
>> > >>>>>>>>>>>>>>>>>>> "google"
>> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
>> > >>>>> finding
>> > >>>>>>> the
>> > >>>>>>>>> right
>> > >>>>>>>>>>>>>>>>>>> answers.
>> > >>>>>>>>>>>>>>>>>>>>>> This
>> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
>> > >>> besides the
>> > >>>>>>>>> community
>> > >>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>> ideas
>> > >>>>>>>>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
>> > >>>>> examples,
>> > >>>>>>>>> problems
>> > >>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>>>> solutions
>> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
>> > >>> remember
>> > >>>>>> that
>> > >>>>>>>>> all the
>> > >>>>>>>>>>>> AI
>> > >>>>>>>>>>>>>>>>>>>>> trained
>> > >>>>>>>>>>>>>>>>>>>>>> on
>> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
>> > >>> matching
>> > >>>>>>> queries
>> > >>>>>>>> of
>> > >>>>>>>>>>>> people.
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
>> > >>>>>> switch.
>> > >>>>>>>>> And if
>> > >>>>>>>>>>> we
>> > >>>>>>>>>>>>>>> do
>> > >>>>>>>>>>>>>>>> -
>> > >>>>>>>>>>>>>>>>>>> I
>> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
>> > >>> instead
>> > >>>>> of
>> > >>>>>>>>> `dag` if
>> > >>>>>>>>>>>> not
>> > >>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>> above
>> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
>> > >>> that it
>> > >>>>>>> might
>> > >>>>>>>>> cause
>> > >>>>>>>>>>>> more
>> > >>>>>>>>>>>>>>>>>>>>> problems
>> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
>> > >>> think
>> > >>>>>>> it's
>> > >>>>>>>> a
>> > >>>>>>>>> good
>> > >>>>>>>>>>>>>>>> idea, I
>> > >>>>>>>>>>>>>>>>>>>>> am
>> > >>>>>>>>>>>>>>>>>>>>>> ok
>> > >>>>>>>>>>>>>>>>>>>>>>> with it.
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> J,
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
>> > >>> Bhakat
>> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
>> > >>> <mailto:
>> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
>> > >>> sense
>> > >>>>> to
>> > >>>>>>>>> someone
>> > >>>>>>>>>>> new
>> > >>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>> workflow
>> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
>> > >>> nature
>> > >>>>>> of
>> > >>>>>>>>> being
>> > >>>>>>>>>>>>>>> acyclic.
>> > >>>>>>>>>>>>>>>>>>>>> Sure,
>> > >>>>>>>>>>>>>>>>>>>>>>> as
>> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
>> > >>> workaround it.
>> > >>>>>>>> Still,
>> > >>>>>>>>> in
>> > >>>>>>>>>>> my
>> > >>>>>>>>>>>>>>>>>>>>> opinion,
>> > >>>>>>>>>>>>>>>>>>>>>>> there
>> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
>> > >>>>>>> workflow
>> > >>>>>>>>>>>>>>>>>>> orchestration.
>> > >>>>>>>>>>>>>>>>>>>>>> Most
>> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
>> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
>> > >>>>> covered
>> > >>>>>>>>> using an
>> > >>>>>>>>>>>>>>>> acyclic
>> > >>>>>>>>>>>>>>>>>>>>>> manner
>> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
>> > >>> idempotency. So
>> > >>>>> I
>> > >>>>>>>> would
>> > >>>>>>>>> want
>> > >>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
>> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
>> > >>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
>> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
>> > >>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
>> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
>> > >>> bernd.stroehle@kosakya.de>>
>> > >>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
>> > >>>>> Airflow
>> > >>>>>>> Fan;
>> > >>>>>>>>> so
>> > >>>>>>>>>>> many
>> > >>>>>>>>>>>>>>> new
>> > >>>>>>>>>>>>>>>>>>>>>> ideas.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
>> > >>>>>> replaced
>> > >>>>>>> by
>> > >>>>>>>>> the
>> > >>>>>>>>>>>> more
>> > >>>>>>>>>>>>>>>>>>>>> general
>> > >>>>>>>>>>>>>>>>>>>>>>>> Term
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
>> > >>>>>> (Petri)
>> > >>>>>>>> Net
>> > >>>>>>>>>>> (AFN)
>> > >>>>>>>>>>>>>>>>>>> (maybe
>> > >>>>>>>>>>>>>>>>>>>>>>>> without
>> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
>> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
>> > >>>>> Graph
>> > >>>>>>>>> Database.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
>> > >>> Graph
>> > >>>>>> (AFG)
>> > >>>>>>>>> might be
>> > >>>>>>>>>>>>>>>>>>>>> assigned
>> > >>>>>>>>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>>> an
>> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
>> > >>> of a
>> > >>>>>>>> library.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
>> > >>>>> structure
>> > >>>>>>>> than
>> > >>>>>>>>> a
>> > >>>>>>>>>>>>>>>>>>>>> configuration
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
>> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
>> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
>> > >>>>>>>>>>> bernd.stroehle@gmail.com>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
>> > >>> <ikholopov@google.com.INVA
>> > >>>>>>>>> <mailto:
>> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
>> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
>> > >>>>>>>>> dev@airflow.apache.org
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
>> > >>> term
>> > >>>>>>> "DAG"
>> > >>>>>>>>> for
>> > >>>>>>>>>>> end
>> > >>>>>>>>>>>>>>>>>>> users
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
>> > >>>>>> suboptimal,
>> > >>>>>>> it
>> > >>>>>>>>> is
>> > >>>>>>>>>>> part
>> > >>>>>>>>>>>>>>> of
>> > >>>>>>>>>>>>>>>>>>>>>> Airflow
>> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
>> > >>> levels,
>> > >>>>> that
>> > >>>>>>> any
>> > >>>>>>>>>>> attempt
>> > >>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>> change
>> > >>>>>>>>>>>>>>>>>>>>>>> it
>> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
>> > >>> reduce it.
>> > >>>>>> The
>> > >>>>>>>>> only
>> > >>>>>>>>>>>> thing
>> > >>>>>>>>>>>>>>>>>>>>> that is
>> > >>>>>>>>>>>>>>>>>>>>>>>> worse
>> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
>> > >>> when
>> > >>>>>>> there
>> > >>>>>>>>> are
>> > >>>>>>>>>>> two
>> > >>>>>>>>>>>>>>> ways
>> > >>>>>>>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>>>> define
>> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
>> > >>>>> tutorials
>> > >>>>>>>> will
>> > >>>>>>>>>>>> suddenly
>> > >>>>>>>>>>>>>>>>>>>>> become
>> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
>> > >>> as
>> > >>>>>>> "DAG"s.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
>> > >>> users
>> > >>>>> away
>> > >>>>>>>> with
>> > >>>>>>>>> a
>> > >>>>>>>>>>>> number
>> > >>>>>>>>>>>>>>>>>>> of
>> > >>>>>>>>>>>>>>>>>>>>>>>> breaking
>> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
>> > >>>>>> breaking
>> > >>>>>>>>> changes
>> > >>>>>>>>>>>> for
>> > >>>>>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>>> most
>> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
>> > >>> are
>> > >>>>>>>> looking
>> > >>>>>>>>> for.
>> > >>>>>>>>>>>>>>>>>>>>> Attempting
>> > >>>>>>>>>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
>> > >>>>> interpreted
>> > >>>>>>> as
>> > >>>>>>>> an
>> > >>>>>>>>>>> even
>> > >>>>>>>>>>>>>>>>>>>>> stronger
>> > >>>>>>>>>>>>>>>>>>>>>>>> signal
>> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
>> > >>> the
>> > >>>>>> term
>> > >>>>>>>> in
>> > >>>>>>>>> the
>> > >>>>>>>>>>>> long
>> > >>>>>>>>>>>>>>>>>>>>> run. I
>> > >>>>>>>>>>>>>>>>>>>>>>>> even
>> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
>> > >>> it in
>> > >>>>>> the
>> > >>>>>>>> DAG
>> > >>>>>>>>>>>>>>> definition
>> > >>>>>>>>>>>>>>>>>>>>>>>> interface.
>> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
>> > >>> other
>> > >>>>>> places
>> > >>>>>>>> in
>> > >>>>>>>>>>>> Airflow,
>> > >>>>>>>>>>>>>>>>>>> of
>> > >>>>>>>>>>>>>>>>>>>>>> which
>> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
>> > >>> Harenslak
>> > >>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
>> > >>>>>>>> bas@astronomer.io.inva
>> > >>>>>>>>>> lid
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
>> > >>> have
>> > >>>>>>> already
>> > >>>>>>>>> faded
>> > >>>>>>>>>>>> over
>> > >>>>>>>>>>>>>>>>>>>>> time,
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
>> > >>>>>> create
>> > >>>>>>>>> cyclic
>> > >>>>>>>>>>>>>>>>>>> graphs,
>> > >>>>>>>>>>>>>>>>>>>>>> e.g.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
>> > >>>>> these
>> > >>>>>>>>>>> properties
>> > >>>>>>>>>>>>>>>>>>>>>> vanishing
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
>> > >>>>>>> perspective I
>> > >>>>>>>>>>> support
>> > >>>>>>>>>>>>>>>>>>>>>> changing
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
>> > >>>>>> naming:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
>> > >>> name.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
>> > >>> change, I
>> > >>>>>>> suggest
>> > >>>>>>>>>>>> defining
>> > >>>>>>>>>>>>>>> a
>> > >>>>>>>>>>>>>>>>>>>>>> clear
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
>> > >>> the
>> > >>>>>>>>> deprecation in
>> > >>>>>>>>>>>>>>>>>>>>> Airflow
>> > >>>>>>>>>>>>>>>>>>>>>> 3,
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
>> > >>>>>>>> neil4real@gmail.com
>> > >>>>>>>>>>> <ma...@gmail.com>>
>> > >>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
>> > >>>>>>>> especially
>> > >>>>>>>>> when
>> > >>>>>>>>>>>>>>>>>>> most
>> > >>>>>>>>>>>>>>>>>>>>>> other
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
>> > >>> wholeheartedly.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
>> > >>>>> confusing
>> > >>>>>>>>> about it
>> > >>>>>>>>>>> at
>> > >>>>>>>>>>>>>>>>>>>>> all,
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
>> > >>>>>>> confusing
>> > >>>>>>>>> to
>> > >>>>>>>>>>> most
>> > >>>>>>>>>>>>>>>>>>>>> users
>> > >>>>>>>>>>>>>>>>>>>>>>> who
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
>> > >>>>> Chung
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
>> > >>>>>>>>> tp@astronomer.io.inva
>> > >>>>>>>>>>>> lid
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
>> > >>> the
>> > >>>>> term
>> > >>>>>>>> DAG.
>> > >>>>>>>>> The
>> > >>>>>>>>>>>> only
>> > >>>>>>>>>>>>>>>>>>>>>>> problem
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
>> > >>>>>>> myselfâto
>> > >>>>>>>>> stop
>> > >>>>>>>>>>>> using
>> > >>>>>>>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
>> > >>> mentioned
>> > >>>>>>>>> âworkflowâ is
>> > >>>>>>>>>>>> not
>> > >>>>>>>>>>>>>>>>>>>>> very
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
>> > >>>>>>> proposed,
>> > >>>>>>>>> but
>> > >>>>>>>>>>>> itâs a
>> > >>>>>>>>>>>>>>>>>>>>> bit
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
>> > >>> places
>> > >>>>> and
>> > >>>>>>> kept
>> > >>>>>>>>>>> finding
>> > >>>>>>>>>>>>>>>>>>>>> myself
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
>> > >>> wanting to
>> > >>>>>>>> clarify
>> > >>>>>>>>> âDAG
>> > >>>>>>>>>>>>>>>>>>>>>>> definitionâ
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
>> > >>> Scheffler
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
>> > >>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
>> > >>> exactly
>> > >>>>> same
>> > >>>>>>>>>>>> observation,
>> > >>>>>>>>>>>>>>>>>>>>> had a
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
>> > >>> always
>> > >>>>> an
>> > >>>>>>>>>>> introduction
>> > >>>>>>>>>>>>>>>>>>> if
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
>> > >>> makes
>> > >>>>>>> sense.
>> > >>>>>>>>>>>>>>>>>>> Especially
>> > >>>>>>>>>>>>>>>>>>>>>> when
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> also
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
>> > >>>>>>> reasonable
>> > >>>>>>>>> step.
>> > >>>>>>>>>>>>>>>>>>>>> Concepts
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> can
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
>> > >>>>> below
>> > >>>>>>> the
>> > >>>>>>>>> desk
>> > >>>>>>>>>>>> with
>> > >>>>>>>>>>>>>>>>>>>>> you
>> > >>>>>>>>>>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
>> > >>> keep
>> > >>>>>>>> details
>> > >>>>>>>>> of
>> > >>>>>>>>>>>>>>>>>>> python
>> > >>>>>>>>>>>>>>>>>>>>>> names
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
>> > >>> DAGâŠ
>> > >>>>>> but
>> > >>>>>>>> user
>> > >>>>>>>>>>>> facing
>> > >>>>>>>>>>>>>>>>>>> it
>> > >>>>>>>>>>>>>>>>>>>>>> is a
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
>> > >>> Hatter <
>> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
>> > >>>>>>>>> ryan.hatter@astronomer.io
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
>> > >>> swords... at
>> > >>>>>>> least
>> > >>>>>>>>> for
>> > >>>>>>>>>>>> now.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
>> > >>> meaning to
>> > >>>>>>>> Airflow
>> > >>>>>>>>>>>> users.
>> > >>>>>>>>>>>>>>>>>>>>>> Indeed,
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> has
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
>> > >>>>>> mathematicians
>> > >>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>> software
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
>> > >>>>>> matter.
>> > >>>>>>>> For
>> > >>>>>>>>>>>> someone
>> > >>>>>>>>>>>>>>>>>>>>> new
>> > >>>>>>>>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
>> > >>>>> orchestration,
>> > >>>>>>> one
>> > >>>>>>>>> of
>> > >>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>> first
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
>> > >>> DAG?"
>> > >>>>>> The
>> > >>>>>>>>> answer
>> > >>>>>>>>>>> is
>> > >>>>>>>>>>>>>>>>>>>>> almost
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
>> > >>> graph.
>> > >>>>> You
>> > >>>>>>>> don't
>> > >>>>>>>>>>> need
>> > >>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>> worry
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
>> > >>>>> workflow."
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
>> > >>>>> least
>> > >>>>>> a
>> > >>>>>>>>> couple
>> > >>>>>>>>>>>>>>>>>>>>> important
>> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
>> > >>> mentioned
>> > >>>>>>> above,
>> > >>>>>>>>> "DAG"
>> > >>>>>>>>>>>> is
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
>> > >>> confusing.
>> > >>>>>> We
>> > >>>>>>>> want
>> > >>>>>>>>>>>> Airflow
>> > >>>>>>>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>> be
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
>> > >>> off the
>> > >>>>>> bat
>> > >>>>>>>>> creates
>> > >>>>>>>>>>>> an
>> > >>>>>>>>>>>>>>>>>>>>>> initial
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
>> > >>>>>>> Concepts*:
>> > >>>>>>>>> The
>> > >>>>>>>>>>> DAG
>> > >>>>>>>>>>>>>>>>>>> is
>> > >>>>>>>>>>>>>>>>>>>>>> just
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
>> > >>> workflow. The
>> > >>>>>>>>> workflow
>> > >>>>>>>>>>>>>>>>>>> includes
>> > >>>>>>>>>>>>>>>>>>>>>> its
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
>> > >>>>> other
>> > >>>>>>>>>>> parameters,
>> > >>>>>>>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>>>>> other
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
>> > >>> Airflow
>> > >>>>>>>> homepage
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
>> > >>>>>>>>>>> https://airflow.apache.org/>>.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
>> > >>> created by
>> > >>>>>> the
>> > >>>>>>>>>>> community
>> > >>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
>> > >>> workflows.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
>> > >>>>> Airflow?"
>> > >>>>>>> docs
>> > >>>>>>>>> page
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>
>> > >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
>> > >>>>>>>>>>>
>> > >>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html
>> > >>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
>> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
>> > >>> Airflow is
>> > >>>>>>>> without
>> > >>>>>>>>>>> using
>> > >>>>>>>>>>>>>>>>>>>>> "DAG."
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
>> > >>> term
>> > >>>>> is
>> > >>>>>>>>>>> introduced
>> > >>>>>>>>>>>>>>>>>>> out
>> > >>>>>>>>>>>>>>>>>>>>> of
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
>> > >>> explain
>> > >>>>> it:
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
>> > >>>>> collection
>> > >>>>>>> of
>> > >>>>>>>>> tasks
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
>> > >>> in
>> > >>>>>> these
>> > >>>>>>>>>>>>>>>>>>> introductions
>> > >>>>>>>>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
>> > >>>>>> orchestrate
>> > >>>>>>>>> DAGs;
>> > >>>>>>>>>>> it
>> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates
>> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
>> > >>>>>> irrelevant
>> > >>>>>>>> to
>> > >>>>>>>>>>> almost
>> > >>>>>>>>>>>>>>>>>>>>> every
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
>> > >>> alias
>> > >>>>> for
>> > >>>>>>> the
>> > >>>>>>>>> term
>> > >>>>>>>>>>>> "DAG"
>> > >>>>>>>>>>>>>>>>>>>>> and
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
>> > >>> "DAG"
>> > >>>>>> with
>> > >>>>>>>>>>>> "workflow".
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
>> > >>>>> away.
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>
>> > >>> ------------------------------------------------------------------
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
>> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
>> > >>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>> > >>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
>> > >>>>>>>>>>> dev-help@airflow.apache.org>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >
>> >
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
First of all - yes, agree with Lee and Ash, I think we do not need to
change the public interface, I'd also be -0 on that, but docs, internal
strings, yes we could.
But .... After thinking a bit and looking at the discussion here, where
people have concerns, I have an alternative take. And I know this might be
controversial, and some people might find it confusing, but why don't we
come up with a new acronym for DAG, one that will be more airflow and less
"mathematics/algorithm". And I have, I think a good reason for it, and I
even have a proposal. Bear with me,
DAG => *Data Airflow Graph*
As many of us mentioned, DAG and Airflow are almost 100% 1-1 now (with
exception of some niche products - you know which one I speak about - which
are unfortunate enough to use the "Dag" in their name - I will explain why
this is unfortunate, IMHO).
We are lucky to have Airflow for "A" in the acronym and we can make it even
more "synonymous" -> Airflow <> DAG with (A) being Airflow.
Now, why I think "Directed Acyclic Graph" is unfortunate.
Not only because "Directed" and "Acyclic" are mostly not in the vocabulary
of most of our users - as Ryan, starting this thread explained yes, I agree
with it, but also I think there is a more important reason - I simply
believe in the future our DAGs might not (and likely will not) be "acyclic".
I know for some it might be herezy and we have not discussed any of that
yet, but more and more I keep on hearing that Machine Learning Workflows of
the future are often somewhat cyclic. In our case there is one type of
repeat-in-cycle behaviour - we have "task retry on failure" - but this is
only on failure and it basically mens "idempotent" retry - we want to redo
something and get single result out of that and basically discard the
previous result. But machine learning/AI workflows are different. I had a
lot of conversations recently (I even co-organized "AI low-level
engineering and hacking" devroom to learn more about those) - and I keep on
hearing that "Repeat a sequence of things several times and refine" is
precisely what AI / ML workflows will need - especially with Agentic
workflows - often we have cooperating agents which **could** be part of the
same DAG, but currently in Airflow where our DAGs are "Acyclic" we can
only express "repeat several time and refine" as a single task that will do
such refining. If we would like to split such an "agent" doing their own
work in "cycles" and communicating with other "agents" doing the same -
represented as a multi-task-entity, we have no way to express it using our
current "acyclic" semantics.
But - other than our implementation makes that assumption currently - there
is nothing fundamental with Airflow being able to run such "partially
cyclic workflows". We have all the pieces, we would have to change
dependency calculation to allow for cycles, update UI to handle cycles
representation - and we could in theory quite seamlessly, support "cyclic"
workflows.
I know, I know, that's the first time we hear it and that's huge change -
of course, I do not want to start discussing any details of it, not before
Airflow 3.0 is out - but I think we should be open that at some point of
time the "Acyclic" property is going to go away, And my personal intuition
tells me that a year from now we will have acyclic workflows (assuming that
all the foundational work we do on Airflow 3 will really help in decreasing
all the technical dept, and will open our minds to new ideas).
That's why I think - the sooner we get rid of "Directed Acyclic" - the
better. that will - even for us, maintainer free us from being routed in
the acyclic nature of Airflow.
How about we change to "Data Airflow Graph" (or any other good acronym
which has (A)irflow instead of (A)cyclic). The name is just a proposal.
Now.. I go hide somewhere as I believe that might cause a "flamewar" :),
unless on request of Ryan you already sheathed your swords in the
discussion (which seems everyone did).
But honestly - I would really love what you think of "let's just break with
Acyclic NOW" and shape our future by using a different acronym.
J.
On Tue, Feb 18, 2025 at 10:54âŻAM Ephraim Anierobi <ep...@gmail.com>
wrote:
> When I started contributing to Airflow, I had to read up on DAG, and now I
> know what it means. I wonder if we are about to have users who use the term
> Dag without knowing that there's a DAG.
>
> Slightly concerned.
>
> -ephraim
>
> On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com> wrote:
>
> > For user-facing things (e.g., `from airflow import DAG`), Iâm more like
> > -0. But for documentation, docstring, internal things and etc., we
> probably
> > could still change most of them?
> >
> > Best,
> > Wei
> >
> > > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org> wrote:
> > >
> > > I also donât personally think itâs worth the pain (not to mention
> > backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
> > >
> > > -ash
> > >
> > >> On 18 Feb 2025, at 04:40, Daniel Imberman <da...@gmail.com>
> > wrote:
> > >>
> > >> I think my biggest concern is a marketing one and not a technical one.
> > >>
> > >> As has been mentioned on the thread the terms airflow and dag are kind
> > of
> > >> synonymous and I certainly donât want to give the impression that we
> are
> > >> breaking more than we are breaking.
> > >>
> > >> I wouldnât die on this hill, but Iâm slightly concerned.
> > >>
> > >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
> > >>
> > >>> Iâm not sure about adding ruff rules here đ€ I think ruff rules are
> > best
> > >>> suited for user-facing things but not the airflow code base itself.
> If
> > what
> > >>> we mean is adding a rule to avoid users using "DAG" *after* we rename
> > it,
> > >>> it's definitely a +1000.
> > >>>
> > >>> I just created GitHub issues for this removing "DAG"
> > >>>
> > >>> https://github.com/apache/airflow/issues/46842
> > >>> https://github.com/apache/airflow/issues/46843
> > >>>
> > >>> One thing I'm not sure about is whether we want to get rid of `from
> > >>> airflow import DAG` as well. đ€
> > >>>
> > >>> Best,
> > >>> Wei
> > >>>
> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > >>>> Hard to say until it's looked at :)
> > >>>>
> > >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <
> aritrabasu1999@gmail.com
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> I can take it up, it's mostly just a doc update right? Or are we
> > doing
> > >>> code
> > >>>>> files replacement too?
> > >>>>> --
> > >>>>> Regards,
> > >>>>> Aritra Basu
> > >>>>>
> > >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com>
> > wrote:
> > >>>>>
> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any
> volunteers
> > >>> to
> > >>>>>> make it ? It's fun.
> > >>>>>>
> > >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com>
> > >>> wrote:
> > >>>>>>
> > >>>>>>> +1 for ruff rules :)
> > >>>>>>>
> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the
> > >>> Airflow
> > >>>>>>> docs, in line with the new UI changes and to make the renaming
> > >>>>>>> consistent across user-facing pages.
> > >>>>>>>
> > >>>>>>> Regards,
> > >>>>>>> Omkar
> > >>>>>>>
> > >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com>
> > >>> wrote:
> > >>>>>>>
> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> > >>>>>>>>
> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag"
> > >>>>> instead
> > >>>>>>> of
> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the
> > >>>>> codebase
> > >>>>>>> with
> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be
> > >>> đ€),
> > >>>>>>> should
> > >>>>>>>> we
> > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Wei
> > >>>>>>>>>
> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> > >>>>>>>>>> Let me know if that's the direction we want to go.
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
> > >>> synonymous
> > >>>>>> with
> > >>>>>>>>>>> workflows in Airflow through the way we talk and explain
> > >>> what
> > >>>>>>> Airflow
> > >>>>>>>>> is
> > >>>>>>>>>>> all about. Although folks would ask, I donât ever use the
> > >>>>>>>> mathematical
> > >>>>>>>>>>> definition of DAG.
> > >>>>>>>>>>> It will be challenging and possibly confusing for many
> > >>> users
> > >>>>>> making
> > >>>>>>>>> such a
> > >>>>>>>>>>> change - I would rather direct that energy to appending the
> > >>>>>> Oxford
> > >>>>>>>>>>> definition of "dag" to include a reference to workflows in
> > >>>>>> Airflow.
> > >>>>>>>> __
> > >>>>>>>>>>>
> > >>>>>>>>>>> -- Rajesh
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> > >>>>>> <mailto:
> > >>>>>>>>>>> jarek@potiuk.com>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> CAUTION: This email originated from outside of the
> > >>>>> organization.
> > >>>>>> Do
> > >>>>>>>> not
> > >>>>>>>>>>> click links or open attachments unless you can confirm the
> > >>>>> sender
> > >>>>>>> and
> > >>>>>>>>> know
> > >>>>>>>>>>> the content is safe.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > >>>>> expéditeur
> > >>>>>>>>> externe.
> > >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
> > >>> si
> > >>>>> vous
> > >>>>>>> ne
> > >>>>>>>>> pouvez
> > >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
> > >>> pas
> > >>>>>>>> certain
> > >>>>>>>>> que
> > >>>>>>>>>>> le contenu ne présente aucun risque.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> From Guido's post:
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> "Around this time the renaming seems to have been renamed".
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Naming is hard.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> J.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > >>>>> droiddev510e@gmail.com
> > >>>>>>>>> <mailto:
> > >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an
> > >>> Airflow
> > >>>>>> term
> > >>>>>>>> now
> > >>>>>>>>>>>> rather
> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not,
> > >>> I've
> > >>>>>>>> worked
> > >>>>>>>>> with
> > >>>>>>>>>>>> work
> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now
> > >>> and are
> > >>>>>> pro
> > >>>>>>>>> devs
> > >>>>>>>>>>> but
> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For
> > >>> them,
> > >>>>> dag
> > >>>>>> =
> > >>>>>>>>> Airflow.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
> > >>> will be
> > >>>>> a
> > >>>>>>>> major
> > >>>>>>>>>>>> behavior
> > >>>>>>>>>>>> change for the developer community, so we'll need a solid
> > >>>>> plan
> > >>>>>> on
> > >>>>>>>>> how to
> > >>>>>>>>>>>> introduce it, when we do.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> While we discuss this, I'd like to share about the Great
> > >>>>>> Renaming
> > >>>>>>>> in
> > >>>>>>>>> core
> > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably
> > >>> get
> > >>>>>> some
> > >>>>>>>>>>> learnings
> > >>>>>>>>>>>> from there? Who knows!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Guido's blog (2009):
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > >>>>>>>>>>> <
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > >>>>>>>>>
> > >>>>>>>>>>>> Follow-up discussion (2024):
> > >>>>>>>>>>>>
> > >>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082
> > >>>>>>> <
> > >>>>>>>>>>>
> > >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>> Omkar
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> > >>> weilee.rx@gmail.com
> > >>>>>>>>> <mailto:
> > >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I think we should probably just accept it as an airflow
> > >>>>> term.
> > >>>>>>> At
> > >>>>>>>>> least,
> > >>>>>>>>>>>>> thatâs how I understood it when I first used Airflow. I
> > >>>>> feel
> > >>>>>>>>> renaming
> > >>>>>>>>>>> it
> > >>>>>>>>>>>> at
> > >>>>>>>>>>>>> this stage would require considerable effort from
> > >>>>> maintainers
> > >>>>>>> and
> > >>>>>>>>>>>> existing
> > >>>>>>>>>>>>> users without providing equivalent benefits.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Wei
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > >>>>>> kaxilnaik@gmail.com
> > >>>>>>>>>>> <ma...@gmail.com>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
> > >>>>> this
> > >>>>>>>>> discussion
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
> > >>>>>>> disruption
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> > >>>>>>>>> constance@astronomer.io.inva>lid>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
> > >>>>>> experience
> > >>>>>>>>> what a
> > >>>>>>>>>>>> dag
> > >>>>>>>>>>>>> is,
> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes -
> > >>> stuff
> > >>>>>> like
> > >>>>>>>>> dags
> > >>>>>>>>>>>> being
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
> > >>>>>> doesn't
> > >>>>>>>>> come up.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> > >>> embrace
> > >>>>> the
> > >>>>>>> name
> > >>>>>>>>> and
> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> > >>>>> vikram@astronomer.io.inva
> > >>>>>>>>> lid>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> > >>>>> struggling
> > >>>>>>> with
> > >>>>>>>>> this
> > >>>>>>>>>>>>> when I
> > >>>>>>>>>>>>>>>> started working with Airflow.
> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
> > >>>>>>>> established
> > >>>>>>>>>>>> concept
> > >>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>> regardless of the origin.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
> > >>> best
> > >>>>>>>> expressed
> > >>>>>>>>> by
> > >>>>>>>>>>>>> Daniel
> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
> > >>> than
> > >>>>>> the
> > >>>>>>>>> computer
> > >>>>>>>>>>>>>>> science
> > >>>>>>>>>>>>>>>> concept.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>> Vikram
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> > >>>>>> onikolas@amazon.com.inva
> > >>>>>>>>> lid
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
> > >>> right
> > >>>>>> Ryan,
> > >>>>>>>> DAG
> > >>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>> great
> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
> > >>> cause
> > >>>>> much
> > >>>>>>>> more
> > >>>>>>>>>>>> wreckage
> > >>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>> it solves.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
> > >>>>> defining
> > >>>>>>>> DAG.
> > >>>>>>>>> I
> > >>>>>>>>>>>> think
> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
> > >>> for a
> > >>>>>> while
> > >>>>>>>> now
> > >>>>>>>>>>>> anyway,
> > >>>>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>>> that feels like a natural step.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Cheers,
> > >>>>>>>>>>>>>>>>> Niko
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> ________________________________
> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > >>>>>>>>> ash@apache.org>>
> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > >>>>>> dev@airflow.apache.org
> > >>>>>>>>
> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
> > >>> term
> > >>>>>> "DAG"
> > >>>>>>>>> for end
> > >>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
> > >>>>>>>>> organization.
> > >>>>>>>>>>> Do
> > >>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can
> > >>> confirm
> > >>>>>> the
> > >>>>>>>>> sender
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> know
> > >>>>>>>>>>>>>>>>> the content is safe.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> > >>> dâun
> > >>>>>>>>> expéditeur
> > >>>>>>>>>>>>>>> externe.
> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > >>>>>> jointe
> > >>>>>>> si
> > >>>>>>>>> vous
> > >>>>>>>>>>> ne
> > >>>>>>>>>>>>>>>> pouvez
> > >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> > >>> vous
> > >>>>>> nâĂȘtes
> > >>>>>>>> pas
> > >>>>>>>>>>>> certain
> > >>>>>>>>>>>>>>>> que
> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
> > >>> term â
> > >>>>>>>> getting
> > >>>>>>>>> to
> > >>>>>>>>>>>>> re-use
> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
> > >>>>> sides
> > >>>>>>> have
> > >>>>>>>>> good
> > >>>>>>>>>>>>>>> reasons
> > >>>>>>>>>>>>>>>>> presented.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> -a
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
> > >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
> > >>> from,
> > >>>>>>> "well,
> > >>>>>>>>> no
> > >>>>>>>>>>> one
> > >>>>>>>>>>>>>>>>> actually
> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > >>>>>>>>> jarek@potiuk.com
> > >>>>>>>>>>> <ma...@potiuk.com>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
> > >>> the
> > >>>>>>>>>>> tongue-twister,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > >>>>>>>>> jarek@potiuk.com
> > >>>>>>>>>>> <ma...@potiuk.com>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
> > >>> would
> > >>>>> very
> > >>>>>>>> much
> > >>>>>>>>> want
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> able
> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> > >>> further.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
> > >>> it's a
> > >>>>>>>>>>>> tongue-twister,
> > >>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> > >>> pronounce
> > >>>>>> it
> > >>>>>>>>>>> properly.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> J.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> > >>>>>>>> brent@astronomer.io.inva
> > >>>>>>>>>> lid>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
> > >>>>> starting
> > >>>>>> on
> > >>>>>>>>> AIP-38
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> modernize
> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > >>>>>>>>> descriptive of
> > >>>>>>>>>>>> what
> > >>>>>>>>>>>>>>>> one
> > >>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
> > >>> is an
> > >>>>>>>>>>> implementation
> > >>>>>>>>>>>>>>>>> detail.
> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
> > >>> point
> > >>>>>> "DAG"
> > >>>>>>>> has
> > >>>>>>>>>>> become
> > >>>>>>>>>>>>>>>> "dag"
> > >>>>>>>>>>>>>>>>>>> , a
> > >>>>>>>>>>>>>>>>>>>>> word in its own right.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
> > >>>>>> discussion,
> > >>>>>>>>> Airflow
> > >>>>>>>>>>>>>>>> Summit
> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
> > >>>>>> embrace
> > >>>>>>>>> "dag".
> > >>>>>>>>>>> A
> > >>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
> > >>>>> concept
> > >>>>>>>>> behind
> > >>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> word. I
> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> > >>> refactoring so
> > >>>>>>> much
> > >>>>>>>>> code,
> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
> > >>>>> questions,
> > >>>>>>>> etc.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> > >>> Standish
> > >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
> > >>> of
> > >>>>>> pain
> > >>>>>>>> for
> > >>>>>>>>>>>>>>>>> questionable
> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
> > >>> idea. I
> > >>>>>>> agree
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> association
> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
> > >>>>>>> invasive
> > >>>>>>>>>>>>>>> mitigations.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
> > >>>>>> de-emphasize
> > >>>>>>>> the
> > >>>>>>>>>>> math
> > >>>>>>>>>>>>>>> nerd
> > >>>>>>>>>>>>>>>>>>>>> origins
> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> > >>> website /
> > >>>>>> etc,
> > >>>>>>>>> *never
> > >>>>>>>>>>>>>>>> define*
> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
> > >>>>>> graph.
> > >>>>>>>>> Always
> > >>>>>>>>>>>>>>> define
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>> as a
> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
> > >>> like a
> > >>>>>>>>> historical
> > >>>>>>>>>>>>>>>>> footnote,
> > >>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
> > >>> hidden.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
> > >>> docs
> > >>>>> in
> > >>>>>>>>> general
> > >>>>>>>>>>> e.g.
> > >>>>>>>>>>>>>>>>>>> writing
> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> > >>> "DAGs"
> > >>>>>> etc.
> > >>>>>>>> The
> > >>>>>>>>>>>> upper
> > >>>>>>>>>>>>>>>> case
> > >>>>>>>>>>>>>>>>>>>>> part
> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
> > >>> "dag" in
> > >>>>>>>>> airlfow is
> > >>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
> > >>> "DAGs"
> > >>>>> is
> > >>>>>>> not
> > >>>>>>>>> really
> > >>>>>>>>>>>>>>>>>>> unhelpful.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
> > >>> is
> > >>>>> its
> > >>>>>>> own
> > >>>>>>>>>>> thing,
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
> > >>> nobody
> > >>>>>> knows
> > >>>>>>>>> about
> > >>>>>>>>>>>>>>>> anyway),
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
> > >>> normal
> > >>>>>>> people
> > >>>>>>>>>>>>>>> understand.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
> > >>> <
> > >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
> > >>> will
> > >>>>> be
> > >>>>>>>>> extremely
> > >>>>>>>>>>>>>>>>>>>>> difficult to
> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
> > >>> will
> > >>>>>> make
> > >>>>>>> a
> > >>>>>>>>> lot of
> > >>>>>>>>>>>>>>>>>>> "google"
> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
> > >>>>> finding
> > >>>>>>> the
> > >>>>>>>>> right
> > >>>>>>>>>>>>>>>>>>> answers.
> > >>>>>>>>>>>>>>>>>>>>>> This
> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> > >>> besides the
> > >>>>>>>>> community
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> ideas
> > >>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
> > >>>>> examples,
> > >>>>>>>>> problems
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> solutions
> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> > >>> remember
> > >>>>>> that
> > >>>>>>>>> all the
> > >>>>>>>>>>>> AI
> > >>>>>>>>>>>>>>>>>>>>> trained
> > >>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> > >>> matching
> > >>>>>>> queries
> > >>>>>>>> of
> > >>>>>>>>>>>> people.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
> > >>>>>> switch.
> > >>>>>>>>> And if
> > >>>>>>>>>>> we
> > >>>>>>>>>>>>>>> do
> > >>>>>>>>>>>>>>>> -
> > >>>>>>>>>>>>>>>>>>> I
> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
> > >>> instead
> > >>>>> of
> > >>>>>>>>> `dag` if
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> above
> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
> > >>> that it
> > >>>>>>> might
> > >>>>>>>>> cause
> > >>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>> problems
> > >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
> > >>> think
> > >>>>>>> it's
> > >>>>>>>> a
> > >>>>>>>>> good
> > >>>>>>>>>>>>>>>> idea, I
> > >>>>>>>>>>>>>>>>>>>>> am
> > >>>>>>>>>>>>>>>>>>>>>> ok
> > >>>>>>>>>>>>>>>>>>>>>>> with it.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> J,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> > >>> Bhakat
> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> > >>> <mailto:
> > >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
> > >>> sense
> > >>>>> to
> > >>>>>>>>> someone
> > >>>>>>>>>>> new
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> workflow
> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
> > >>> nature
> > >>>>>> of
> > >>>>>>>>> being
> > >>>>>>>>>>>>>>> acyclic.
> > >>>>>>>>>>>>>>>>>>>>> Sure,
> > >>>>>>>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> > >>> workaround it.
> > >>>>>>>> Still,
> > >>>>>>>>> in
> > >>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>> opinion,
> > >>>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
> > >>>>>>> workflow
> > >>>>>>>>>>>>>>>>>>> orchestration.
> > >>>>>>>>>>>>>>>>>>>>>> Most
> > >>>>>>>>>>>>>>>>>>>>>>>> (*if
> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
> > >>>>> covered
> > >>>>>>>>> using an
> > >>>>>>>>>>>>>>>> acyclic
> > >>>>>>>>>>>>>>>>>>>>>> manner
> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> > >>> idempotency. So
> > >>>>> I
> > >>>>>>>> would
> > >>>>>>>>> want
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Avi
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> > >>> bernd.stroehle@kosakya.de>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
> > >>>>> Airflow
> > >>>>>>> Fan;
> > >>>>>>>>> so
> > >>>>>>>>>>> many
> > >>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>> ideas.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
> > >>>>>> replaced
> > >>>>>>> by
> > >>>>>>>>> the
> > >>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>> general
> > >>>>>>>>>>>>>>>>>>>>>>>> Term
> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> > >>>>>> (Petri)
> > >>>>>>>> Net
> > >>>>>>>>>>> (AFN)
> > >>>>>>>>>>>>>>>>>>> (maybe
> > >>>>>>>>>>>>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
> > >>>>> Graph
> > >>>>>>>>> Database.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> > >>> Graph
> > >>>>>> (AFG)
> > >>>>>>>>> might be
> > >>>>>>>>>>>>>>>>>>>>> assigned
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
> > >>> of a
> > >>>>>>>> library.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
> > >>>>> structure
> > >>>>>>>> than
> > >>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>> configuration
> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > >>>>>>>>>>> bernd.stroehle@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> > >>> <ikholopov@google.com.INVA
> > >>>>>>>>> <mailto:
> > >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > >>>>>>>>> dev@airflow.apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
> > >>> term
> > >>>>>>> "DAG"
> > >>>>>>>>> for
> > >>>>>>>>>>> end
> > >>>>>>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
> > >>>>>> suboptimal,
> > >>>>>>> it
> > >>>>>>>>> is
> > >>>>>>>>>>> part
> > >>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>> Airflow
> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
> > >>> levels,
> > >>>>> that
> > >>>>>>> any
> > >>>>>>>>>>> attempt
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> change
> > >>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
> > >>> reduce it.
> > >>>>>> The
> > >>>>>>>>> only
> > >>>>>>>>>>>> thing
> > >>>>>>>>>>>>>>>>>>>>> that is
> > >>>>>>>>>>>>>>>>>>>>>>>> worse
> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
> > >>> when
> > >>>>>>> there
> > >>>>>>>>> are
> > >>>>>>>>>>> two
> > >>>>>>>>>>>>>>> ways
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>> define
> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
> > >>>>> tutorials
> > >>>>>>>> will
> > >>>>>>>>>>>> suddenly
> > >>>>>>>>>>>>>>>>>>>>> become
> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
> > >>> as
> > >>>>>>> "DAG"s.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
> > >>> users
> > >>>>> away
> > >>>>>>>> with
> > >>>>>>>>> a
> > >>>>>>>>>>>> number
> > >>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>> breaking
> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
> > >>>>>> breaking
> > >>>>>>>>> changes
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
> > >>> are
> > >>>>>>>> looking
> > >>>>>>>>> for.
> > >>>>>>>>>>>>>>>>>>>>> Attempting
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
> > >>>>> interpreted
> > >>>>>>> as
> > >>>>>>>> an
> > >>>>>>>>>>> even
> > >>>>>>>>>>>>>>>>>>>>> stronger
> > >>>>>>>>>>>>>>>>>>>>>>>> signal
> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
> > >>> the
> > >>>>>> term
> > >>>>>>>> in
> > >>>>>>>>> the
> > >>>>>>>>>>>> long
> > >>>>>>>>>>>>>>>>>>>>> run. I
> > >>>>>>>>>>>>>>>>>>>>>>>> even
> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
> > >>> it in
> > >>>>>> the
> > >>>>>>>> DAG
> > >>>>>>>>>>>>>>> definition
> > >>>>>>>>>>>>>>>>>>>>>>>> interface.
> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
> > >>> other
> > >>>>>> places
> > >>>>>>>> in
> > >>>>>>>>>>>> Airflow,
> > >>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
> > >>> Harenslak
> > >>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > >>>>>>>> bas@astronomer.io.inva
> > >>>>>>>>>> lid
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
> > >>> have
> > >>>>>>> already
> > >>>>>>>>> faded
> > >>>>>>>>>>>> over
> > >>>>>>>>>>>>>>>>>>>>> time,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
> > >>>>>> create
> > >>>>>>>>> cyclic
> > >>>>>>>>>>>>>>>>>>> graphs,
> > >>>>>>>>>>>>>>>>>>>>>> e.g.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
> > >>>>> these
> > >>>>>>>>>>> properties
> > >>>>>>>>>>>>>>>>>>>>>> vanishing
> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
> > >>>>>>> perspective I
> > >>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>> changing
> > >>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
> > >>>>>> naming:
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
> > >>> name.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
> > >>> change, I
> > >>>>>>> suggest
> > >>>>>>>>>>>> defining
> > >>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>> clear
> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
> > >>> the
> > >>>>>>>>> deprecation in
> > >>>>>>>>>>>>>>>>>>>>> Airflow
> > >>>>>>>>>>>>>>>>>>>>>> 3,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > >>>>>>>> neil4real@gmail.com
> > >>>>>>>>>>> <ma...@gmail.com>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > >>>>>>>> especially
> > >>>>>>>>> when
> > >>>>>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
> > >>> wholeheartedly.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
> > >>>>> confusing
> > >>>>>>>>> about it
> > >>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>>>>>> all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
> > >>>>>>> confusing
> > >>>>>>>>> to
> > >>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>> users
> > >>>>>>>>>>>>>>>>>>>>>>> who
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been
> > >>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> > >>>>> Chung
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > >>>>>>>>> tp@astronomer.io.inva
> > >>>>>>>>>>>> lid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
> > >>> the
> > >>>>> term
> > >>>>>>>> DAG.
> > >>>>>>>>> The
> > >>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>>>>>>> problem
> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
> > >>>>>>> myselfâto
> > >>>>>>>>> stop
> > >>>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
> > >>> mentioned
> > >>>>>>>>> âworkflowâ is
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>> very
> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> > >>>>>>> proposed,
> > >>>>>>>>> but
> > >>>>>>>>>>>> itâs a
> > >>>>>>>>>>>>>>>>>>>>> bit
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
> > >>> places
> > >>>>> and
> > >>>>>>> kept
> > >>>>>>>>>>> finding
> > >>>>>>>>>>>>>>>>>>>>> myself
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
> > >>> wanting to
> > >>>>>>>> clarify
> > >>>>>>>>> âDAG
> > >>>>>>>>>>>>>>>>>>>>>>> definitionâ
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
> > >>> Scheffler
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > >>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
> > >>> exactly
> > >>>>> same
> > >>>>>>>>>>>> observation,
> > >>>>>>>>>>>>>>>>>>>>> had a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
> > >>> always
> > >>>>> an
> > >>>>>>>>>>> introduction
> > >>>>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
> > >>> makes
> > >>>>>>> sense.
> > >>>>>>>>>>>>>>>>>>> Especially
> > >>>>>>>>>>>>>>>>>>>>>> when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> > >>>>>>> reasonable
> > >>>>>>>>> step.
> > >>>>>>>>>>>>>>>>>>>>> Concepts
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still
> > >>>>>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> > >>>>> below
> > >>>>>>> the
> > >>>>>>>>> desk
> > >>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
> > >>> keep
> > >>>>>>>> details
> > >>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>> python
> > >>>>>>>>>>>>>>>>>>>>>> names
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
> > >>> DAGâŠ
> > >>>>>> but
> > >>>>>>>> user
> > >>>>>>>>>>>> facing
> > >>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
> > >>> Hatter <
> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > >>>>>>>>> ryan.hatter@astronomer.io
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
> > >>> swords... at
> > >>>>>>> least
> > >>>>>>>>> for
> > >>>>>>>>>>>> now.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
> > >>> meaning to
> > >>>>>>>> Airflow
> > >>>>>>>>>>>> users.
> > >>>>>>>>>>>>>>>>>>>>>> Indeed,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
> > >>>>>> mathematicians
> > >>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> software
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> > >>>>>> matter.
> > >>>>>>>> For
> > >>>>>>>>>>>> someone
> > >>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
> > >>>>> orchestration,
> > >>>>>>> one
> > >>>>>>>>> of
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> first
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
> > >>> DAG?"
> > >>>>>> The
> > >>>>>>>>> answer
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> almost
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
> > >>> graph.
> > >>>>> You
> > >>>>>>>> don't
> > >>>>>>>>>>> need
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> worry
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
> > >>>>> workflow."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> > >>>>> least
> > >>>>>> a
> > >>>>>>>>> couple
> > >>>>>>>>>>>>>>>>>>>>> important
> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
> > >>> mentioned
> > >>>>>>> above,
> > >>>>>>>>> "DAG"
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
> > >>> confusing.
> > >>>>>> We
> > >>>>>>>> want
> > >>>>>>>>>>>> Airflow
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
> > >>> off the
> > >>>>>> bat
> > >>>>>>>>> creates
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> > >>>>>>> Concepts*:
> > >>>>>>>>> The
> > >>>>>>>>>>> DAG
> > >>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
> > >>> workflow. The
> > >>>>>>>>> workflow
> > >>>>>>>>>>>>>>>>>>> includes
> > >>>>>>>>>>>>>>>>>>>>>> its
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> > >>>>> other
> > >>>>>>>>>>> parameters,
> > >>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
> > >>> Airflow
> > >>>>>>>> homepage
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > >>>>>>>>>>> https://airflow.apache.org/>>.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
> > >>> created by
> > >>>>>> the
> > >>>>>>>>>>> community
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
> > >>> workflows.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
> > >>>>> Airflow?"
> > >>>>>>> docs
> > >>>>>>>>> page
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>
> > >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > >>>>>>>>>>>
> > >>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > >>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
> > >>> Airflow is
> > >>>>>>>> without
> > >>>>>>>>>>> using
> > >>>>>>>>>>>>>>>>>>>>> "DAG."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
> > >>> term
> > >>>>> is
> > >>>>>>>>>>> introduced
> > >>>>>>>>>>>>>>>>>>> out
> > >>>>>>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
> > >>> explain
> > >>>>> it:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> > >>>>> collection
> > >>>>>>> of
> > >>>>>>>>> tasks
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
> > >>> in
> > >>>>>> these
> > >>>>>>>>>>>>>>>>>>> introductions
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> > >>>>>> orchestrate
> > >>>>>>>>> DAGs;
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates
> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> > >>>>>> irrelevant
> > >>>>>>>> to
> > >>>>>>>>>>> almost
> > >>>>>>>>>>>>>>>>>>>>> every
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
> > >>> alias
> > >>>>> for
> > >>>>>>> the
> > >>>>>>>>> term
> > >>>>>>>>>>>> "DAG"
> > >>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
> > >>> "DAG"
> > >>>>>> with
> > >>>>>>>>>>>> "workflow".
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
> > >>>>> away.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>
> > >>> ------------------------------------------------------------------
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > >>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> > >>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
> > >>>>>>>>>>> dev-help@airflow.apache.org>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Ephraim Anierobi <ep...@gmail.com>.
When I started contributing to Airflow, I had to read up on DAG, and now I
know what it means. I wonder if we are about to have users who use the term
Dag without knowing that there's a DAG.
Slightly concerned.
-ephraim
On Tue, 18 Feb 2025 at 10:26, Wei Lee <we...@gmail.com> wrote:
> For user-facing things (e.g., `from airflow import DAG`), Iâm more like
> -0. But for documentation, docstring, internal things and etc., we probably
> could still change most of them?
>
> Best,
> Wei
>
> > On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > I also donât personally think itâs worth the pain (not to mention
> backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
> >
> > -ash
> >
> >> On 18 Feb 2025, at 04:40, Daniel Imberman <da...@gmail.com>
> wrote:
> >>
> >> I think my biggest concern is a marketing one and not a technical one.
> >>
> >> As has been mentioned on the thread the terms airflow and dag are kind
> of
> >> synonymous and I certainly donât want to give the impression that we are
> >> breaking more than we are breaking.
> >>
> >> I wouldnât die on this hill, but Iâm slightly concerned.
> >>
> >> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
> >>
> >>> Iâm not sure about adding ruff rules here đ€ I think ruff rules are
> best
> >>> suited for user-facing things but not the airflow code base itself. If
> what
> >>> we mean is adding a rule to avoid users using "DAG" *after* we rename
> it,
> >>> it's definitely a +1000.
> >>>
> >>> I just created GitHub issues for this removing "DAG"
> >>>
> >>> https://github.com/apache/airflow/issues/46842
> >>> https://github.com/apache/airflow/issues/46843
> >>>
> >>> One thing I'm not sure about is whether we want to get rid of `from
> >>> airflow import DAG` as well. đ€
> >>>
> >>> Best,
> >>> Wei
> >>>
> >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> >>>> Hard to say until it's looked at :)
> >>>>
> >>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <aritrabasu1999@gmail.com
> >
> >>>> wrote:
> >>>>
> >>>>> I can take it up, it's mostly just a doc update right? Or are we
> doing
> >>> code
> >>>>> files replacement too?
> >>>>> --
> >>>>> Regards,
> >>>>> Aritra Basu
> >>>>>
> >>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com>
> wrote:
> >>>>>
> >>>>>> Sounds like another +1000 files big PR is coming :) ? Any volunteers
> >>> to
> >>>>>> make it ? It's fun.
> >>>>>>
> >>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>>> +1 for ruff rules :)
> >>>>>>>
> >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the
> >>> Airflow
> >>>>>>> docs, in line with the new UI changes and to make the renaming
> >>>>>>> consistent across user-facing pages.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Omkar
> >>>>>>>
> >>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
> >>>>>>>>
> >>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag"
> >>>>> instead
> >>>>>>> of
> >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the
> >>>>> codebase
> >>>>>>> with
> >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be
> >>> đ€),
> >>>>>>> should
> >>>>>>>> we
> >>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Wei
> >>>>>>>>>
> >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
> >>>>>>>>>> https://github.com/apache/airflow/pull/43325
> >>>>>>>>>> Let me know if that's the direction we want to go.
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> >>>>>>>>>> <rb...@amazon.com.invalid> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
> >>> synonymous
> >>>>>> with
> >>>>>>>>>>> workflows in Airflow through the way we talk and explain
> >>> what
> >>>>>>> Airflow
> >>>>>>>>> is
> >>>>>>>>>>> all about. Although folks would ask, I donât ever use the
> >>>>>>>> mathematical
> >>>>>>>>>>> definition of DAG.
> >>>>>>>>>>> It will be challenging and possibly confusing for many
> >>> users
> >>>>>> making
> >>>>>>>>> such a
> >>>>>>>>>>> change - I would rather direct that energy to appending the
> >>>>>> Oxford
> >>>>>>>>>>> definition of "dag" to include a reference to workflows in
> >>>>>> Airflow.
> >>>>>>>> __
> >>>>>>>>>>>
> >>>>>>>>>>> -- Rajesh
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> >>>>>> <mailto:
> >>>>>>>>>>> jarek@potiuk.com>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> CAUTION: This email originated from outside of the
> >>>>> organization.
> >>>>>> Do
> >>>>>>>> not
> >>>>>>>>>>> click links or open attachments unless you can confirm the
> >>>>> sender
> >>>>>>> and
> >>>>>>>>> know
> >>>>>>>>>>> the content is safe.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> >>>>> expéditeur
> >>>>>>>>> externe.
> >>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
> >>> si
> >>>>> vous
> >>>>>>> ne
> >>>>>>>>> pouvez
> >>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
> >>> pas
> >>>>>>>> certain
> >>>>>>>>> que
> >>>>>>>>>>> le contenu ne présente aucun risque.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> From Guido's post:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> "Around this time the renaming seems to have been renamed".
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Naming is hard.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> J.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> >>>>> droiddev510e@gmail.com
> >>>>>>>>> <mailto:
> >>>>>>>>>>> droiddev510e@gmail.com>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> Yes, completely agree with above comments, dag is an
> >>> Airflow
> >>>>>> term
> >>>>>>>> now
> >>>>>>>>>>>> rather
> >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not,
> >>> I've
> >>>>>>>> worked
> >>>>>>>>> with
> >>>>>>>>>>>> work
> >>>>>>>>>>>> folks who've been using dags in Airflow for years now
> >>> and are
> >>>>>> pro
> >>>>>>>>> devs
> >>>>>>>>>>> but
> >>>>>>>>>>>> have trouble remembering the full form of a DAG! For
> >>> them,
> >>>>> dag
> >>>>>> =
> >>>>>>>>> Airflow.
> >>>>>>>>>>>>
> >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
> >>> will be
> >>>>> a
> >>>>>>>> major
> >>>>>>>>>>>> behavior
> >>>>>>>>>>>> change for the developer community, so we'll need a solid
> >>>>> plan
> >>>>>> on
> >>>>>>>>> how to
> >>>>>>>>>>>> introduce it, when we do.
> >>>>>>>>>>>>
> >>>>>>>>>>>> While we discuss this, I'd like to share about the Great
> >>>>>> Renaming
> >>>>>>>> in
> >>>>>>>>> core
> >>>>>>>>>>>> Python started by Guido back in 2009. We could probably
> >>> get
> >>>>>> some
> >>>>>>>>>>> learnings
> >>>>>>>>>>>> from there? Who knows!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Guido's blog (2009):
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> >>>>>>>>>>> <
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> >>>>>>>>>
> >>>>>>>>>>>> Follow-up discussion (2024):
> >>>>>>>>>>>>
> >>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082
> >>>>>>> <
> >>>>>>>>>>>
> >>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Omkar
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> >>> weilee.rx@gmail.com
> >>>>>>>>> <mailto:
> >>>>>>>>>>> weilee.rx@gmail.com>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I think we should probably just accept it as an airflow
> >>>>> term.
> >>>>>>> At
> >>>>>>>>> least,
> >>>>>>>>>>>>> thatâs how I understood it when I first used Airflow. I
> >>>>> feel
> >>>>>>>>> renaming
> >>>>>>>>>>> it
> >>>>>>>>>>>> at
> >>>>>>>>>>>>> this stage would require considerable effort from
> >>>>> maintainers
> >>>>>>> and
> >>>>>>>>>>>> existing
> >>>>>>>>>>>>> users without providing equivalent benefits.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Wei
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> >>>>>> kaxilnaik@gmail.com
> >>>>>>>>>>> <ma...@gmail.com>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
> >>>>> this
> >>>>>>>>> discussion
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
> >>>>>>> disruption
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> >>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
> >>>>>>>>> constance@astronomer.io.inva>lid>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
> >>>>>> experience
> >>>>>>>>> what a
> >>>>>>>>>>>> dag
> >>>>>>>>>>>>> is,
> >>>>>>>>>>>>>>> they'll start talking about workflow attributes -
> >>> stuff
> >>>>>> like
> >>>>>>>>> dags
> >>>>>>>>>>>> being
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
> >>>>>> doesn't
> >>>>>>>>> come up.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
> >>> embrace
> >>>>> the
> >>>>>>> name
> >>>>>>>>> and
> >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> >>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
> >>>>> vikram@astronomer.io.inva
> >>>>>>>>> lid>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It's an interesting discussion and I remember
> >>>>> struggling
> >>>>>>> with
> >>>>>>>>> this
> >>>>>>>>>>>>> when I
> >>>>>>>>>>>>>>>> started working with Airflow.
> >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
> >>>>>>>> established
> >>>>>>>>>>>> concept
> >>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>> regardless of the origin.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I am personally leaning towards the perspective
> >>> best
> >>>>>>>> expressed
> >>>>>>>>> by
> >>>>>>>>>>>>> Daniel
> >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
> >>> than
> >>>>>> the
> >>>>>>>>> computer
> >>>>>>>>>>>>>>> science
> >>>>>>>>>>>>>>>> concept.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>> Vikram
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> >>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
> >>>>>> onikolas@amazon.com.inva
> >>>>>>>>> lid
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
> >>> right
> >>>>>> Ryan,
> >>>>>>>> DAG
> >>>>>>>>>>> isn't
> >>>>>>>>>>>>>>> great
> >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
> >>> cause
> >>>>> much
> >>>>>>>> more
> >>>>>>>>>>>> wreckage
> >>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>> it solves.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
> >>>>> defining
> >>>>>>>> DAG.
> >>>>>>>>> I
> >>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>> we've been naturally doing that as a community
> >>> for a
> >>>>>> while
> >>>>>>>> now
> >>>>>>>>>>>> anyway,
> >>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>> that feels like a natural step.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>> Niko
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ________________________________
> >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> >>>>>>>>> ash@apache.org>>
> >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> >>>>>> dev@airflow.apache.org
> >>>>>>>>
> >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
> >>> term
> >>>>>> "DAG"
> >>>>>>>>> for end
> >>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
> >>>>>>>>> organization.
> >>>>>>>>>>> Do
> >>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> click links or open attachments unless you can
> >>> confirm
> >>>>>> the
> >>>>>>>>> sender
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> know
> >>>>>>>>>>>>>>>>> the content is safe.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
> >>> dâun
> >>>>>>>>> expéditeur
> >>>>>>>>>>>>>>> externe.
> >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> >>>>>> jointe
> >>>>>>> si
> >>>>>>>>> vous
> >>>>>>>>>>> ne
> >>>>>>>>>>>>>>>> pouvez
> >>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> >>> vous
> >>>>>> nâĂȘtes
> >>>>>>>> pas
> >>>>>>>>>>>> certain
> >>>>>>>>>>>>>>>> que
> >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
> >>> term â
> >>>>>>>> getting
> >>>>>>>>> to
> >>>>>>>>>>>>> re-use
> >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
> >>>>> sides
> >>>>>>> have
> >>>>>>>>> good
> >>>>>>>>>>>>>>> reasons
> >>>>>>>>>>>>>>>>> presented.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -a
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> >>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
> >>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
> >>> from,
> >>>>>>> "well,
> >>>>>>>>> no
> >>>>>>>>>>> one
> >>>>>>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>> knows but..." and then make something up.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> >>>>>>>>> jarek@potiuk.com
> >>>>>>>>>>> <ma...@potiuk.com>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
> >>> the
> >>>>>>>>>>> tongue-twister,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> >>>>>>>>> jarek@potiuk.com
> >>>>>>>>>>> <ma...@potiuk.com>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
> >>> would
> >>>>> very
> >>>>>>>> much
> >>>>>>>>> want
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> able
> >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
> >>> further.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
> >>> it's a
> >>>>>>>>>>>> tongue-twister,
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
> >>> pronounce
> >>>>>> it
> >>>>>>>>>>> properly.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> J.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> >>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
> >>>>>>>> brent@astronomer.io.inva
> >>>>>>>>>> lid>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
> >>>>> starting
> >>>>>> on
> >>>>>>>>> AIP-38
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> modernize
> >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
> >>>>>>>>> descriptive of
> >>>>>>>>>>>> what
> >>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
> >>> is an
> >>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>> detail.
> >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
> >>> point
> >>>>>> "DAG"
> >>>>>>>> has
> >>>>>>>>>>> become
> >>>>>>>>>>>>>>>> "dag"
> >>>>>>>>>>>>>>>>>>> , a
> >>>>>>>>>>>>>>>>>>>>> word in its own right.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
> >>>>>> discussion,
> >>>>>>>>> Airflow
> >>>>>>>>>>>>>>>> Summit
> >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
> >>>>>> embrace
> >>>>>>>>> "dag".
> >>>>>>>>>>> A
> >>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
> >>>>> concept
> >>>>>>>>> behind
> >>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> word. I
> >>>>>>>>>>>>>>>>>>>>> think that is much less effort than
> >>> refactoring so
> >>>>>>> much
> >>>>>>>>> code,
> >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
> >>>>> questions,
> >>>>>>>> etc.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> >>> Standish
> >>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> >>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
> >>> of
> >>>>>> pain
> >>>>>>>> for
> >>>>>>>>>>>>>>>>> questionable
> >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
> >>> idea. I
> >>>>>>> agree
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> association
> >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
> >>>>>>> invasive
> >>>>>>>>>>>>>>> mitigations.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
> >>>>>> de-emphasize
> >>>>>>>> the
> >>>>>>>>>>> math
> >>>>>>>>>>>>>>> nerd
> >>>>>>>>>>>>>>>>>>>>> origins
> >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
> >>> website /
> >>>>>> etc,
> >>>>>>>>> *never
> >>>>>>>>>>>>>>>> define*
> >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
> >>>>>> graph.
> >>>>>>>>> Always
> >>>>>>>>>>>>>>> define
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> as a
> >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
> >>> like a
> >>>>>>>>> historical
> >>>>>>>>>>>>>>>>> footnote,
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
> >>> hidden.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
> >>> docs
> >>>>> in
> >>>>>>>>> general
> >>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>>> writing
> >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> >>> "DAGs"
> >>>>>> etc.
> >>>>>>>> The
> >>>>>>>>>>>> upper
> >>>>>>>>>>>>>>>> case
> >>>>>>>>>>>>>>>>>>>>> part
> >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
> >>> "dag" in
> >>>>>>>>> airlfow is
> >>>>>>>>>>>> just
> >>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
> >>> "DAGs"
> >>>>> is
> >>>>>>> not
> >>>>>>>>> really
> >>>>>>>>>>>>>>>>>>> unhelpful.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
> >>> is
> >>>>> its
> >>>>>>> own
> >>>>>>>>>>> thing,
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> *not* strictly
> >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
> >>> nobody
> >>>>>> knows
> >>>>>>>>> about
> >>>>>>>>>>>>>>>> anyway),
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
> >>> normal
> >>>>>>> people
> >>>>>>>>>>>>>>> understand.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
> >>> <
> >>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
> >>> will
> >>>>> be
> >>>>>>>>> extremely
> >>>>>>>>>>>>>>>>>>>>> difficult to
> >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
> >>> will
> >>>>>> make
> >>>>>>> a
> >>>>>>>>> lot of
> >>>>>>>>>>>>>>>>>>> "google"
> >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
> >>>>> finding
> >>>>>>> the
> >>>>>>>>> right
> >>>>>>>>>>>>>>>>>>> answers.
> >>>>>>>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
> >>> besides the
> >>>>>>>>> community
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>> ideas
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
> >>>>> examples,
> >>>>>>>>> problems
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> solutions
> >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
> >>> remember
> >>>>>> that
> >>>>>>>>> all the
> >>>>>>>>>>>> AI
> >>>>>>>>>>>>>>>>>>>>> trained
> >>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
> >>> matching
> >>>>>>> queries
> >>>>>>>> of
> >>>>>>>>>>>> people.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
> >>>>>> switch.
> >>>>>>>>> And if
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
> >>> instead
> >>>>> of
> >>>>>>>>> `dag` if
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> above
> >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
> >>> that it
> >>>>>>> might
> >>>>>>>>> cause
> >>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>> problems
> >>>>>>>>>>>>>>>>>>>>>>> than it solves.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
> >>> think
> >>>>>>> it's
> >>>>>>>> a
> >>>>>>>>> good
> >>>>>>>>>>>>>>>> idea, I
> >>>>>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>>>> ok
> >>>>>>>>>>>>>>>>>>>>>>> with it.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> J,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> >>> Bhakat
> >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> >>> <mailto:
> >>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
> >>> sense
> >>>>> to
> >>>>>>>>> someone
> >>>>>>>>>>> new
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> workflow
> >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
> >>> nature
> >>>>>> of
> >>>>>>>>> being
> >>>>>>>>>>>>>>> acyclic.
> >>>>>>>>>>>>>>>>>>>>> Sure,
> >>>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
> >>> workaround it.
> >>>>>>>> Still,
> >>>>>>>>> in
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>> opinion,
> >>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
> >>>>>>> workflow
> >>>>>>>>>>>>>>>>>>> orchestration.
> >>>>>>>>>>>>>>>>>>>>>> Most
> >>>>>>>>>>>>>>>>>>>>>>>> (*if
> >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
> >>>>> covered
> >>>>>>>>> using an
> >>>>>>>>>>>>>>>> acyclic
> >>>>>>>>>>>>>>>>>>>>>> manner
> >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
> >>> idempotency. So
> >>>>> I
> >>>>>>>> would
> >>>>>>>>> want
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> "acyclic"
> >>>>>>>>>>>>>>>>>>>>>>>> word to stick.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Avi
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> >>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
> >>> bernd.stroehle@kosakya.de>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
> >>>>> Airflow
> >>>>>>> Fan;
> >>>>>>>>> so
> >>>>>>>>>>> many
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>> ideas.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
> >>>>>> replaced
> >>>>>>> by
> >>>>>>>>> the
> >>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>> general
> >>>>>>>>>>>>>>>>>>>>>>>> Term
> >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> >>>>>> (Petri)
> >>>>>>>> Net
> >>>>>>>>>>> (AFN)
> >>>>>>>>>>>>>>>>>>> (maybe
> >>>>>>>>>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>>>>>>>> a direction);
> >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
> >>>>> Graph
> >>>>>>>>> Database.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> >>> Graph
> >>>>>> (AFG)
> >>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>> assigned
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
> >>> of a
> >>>>>>>> library.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
> >>>>> structure
> >>>>>>>> than
> >>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>>> Graph.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
> >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
> >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> >>>>>>>>>>> bernd.stroehle@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
> >>> <ikholopov@google.com.INVA
> >>>>>>>>> <mailto:
> >>>>>>>>>>> ikholopov@google.com.INVA>LID>
> >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> >>>>>>>>> dev@airflow.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
> >>> term
> >>>>>>> "DAG"
> >>>>>>>>> for
> >>>>>>>>>>> end
> >>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
> >>>>>> suboptimal,
> >>>>>>> it
> >>>>>>>>> is
> >>>>>>>>>>> part
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
> >>> levels,
> >>>>> that
> >>>>>>> any
> >>>>>>>>>>> attempt
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
> >>> reduce it.
> >>>>>> The
> >>>>>>>>> only
> >>>>>>>>>>>> thing
> >>>>>>>>>>>>>>>>>>>>> that is
> >>>>>>>>>>>>>>>>>>>>>>>> worse
> >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
> >>> when
> >>>>>>> there
> >>>>>>>>> are
> >>>>>>>>>>> two
> >>>>>>>>>>>>>>> ways
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> define
> >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
> >>>>> tutorials
> >>>>>>>> will
> >>>>>>>>>>>> suddenly
> >>>>>>>>>>>>>>>>>>>>> become
> >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
> >>> as
> >>>>>>> "DAG"s.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
> >>> users
> >>>>> away
> >>>>>>>> with
> >>>>>>>>> a
> >>>>>>>>>>>> number
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> breaking
> >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
> >>>>>> breaking
> >>>>>>>>> changes
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
> >>> are
> >>>>>>>> looking
> >>>>>>>>> for.
> >>>>>>>>>>>>>>>>>>>>> Attempting
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
> >>>>> interpreted
> >>>>>>> as
> >>>>>>>> an
> >>>>>>>>>>> even
> >>>>>>>>>>>>>>>>>>>>> stronger
> >>>>>>>>>>>>>>>>>>>>>>>> signal
> >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
> >>> the
> >>>>>> term
> >>>>>>>> in
> >>>>>>>>> the
> >>>>>>>>>>>> long
> >>>>>>>>>>>>>>>>>>>>> run. I
> >>>>>>>>>>>>>>>>>>>>>>>> even
> >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
> >>> it in
> >>>>>> the
> >>>>>>>> DAG
> >>>>>>>>>>>>>>> definition
> >>>>>>>>>>>>>>>>>>>>>>>> interface.
> >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
> >>> other
> >>>>>> places
> >>>>>>>> in
> >>>>>>>>>>>> Airflow,
> >>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>>>>>>>> Igor
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
> >>> Harenslak
> >>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
> >>>>>>>> bas@astronomer.io.inva
> >>>>>>>>>> lid
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
> >>> have
> >>>>>>> already
> >>>>>>>>> faded
> >>>>>>>>>>>> over
> >>>>>>>>>>>>>>>>>>>>> time,
> >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
> >>>>>> create
> >>>>>>>>> cyclic
> >>>>>>>>>>>>>>>>>>> graphs,
> >>>>>>>>>>>>>>>>>>>>>> e.g.
> >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
> >>>>> these
> >>>>>>>>>>> properties
> >>>>>>>>>>>>>>>>>>>>>> vanishing
> >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
> >>>>>>> perspective I
> >>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>> changing
> >>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
> >>>>>> naming:
> >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
> >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
> >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
> >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
> >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
> >>> name.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
> >>> change, I
> >>>>>>> suggest
> >>>>>>>>>>>> defining
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> clear
> >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
> >>> the
> >>>>>>>>> deprecation in
> >>>>>>>>>>>>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>>> 3,
> >>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Bas
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> >>>>>>>> neil4real@gmail.com
> >>>>>>>>>>> <ma...@gmail.com>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
> >>>>>>>> especially
> >>>>>>>>> when
> >>>>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
> >>> wholeheartedly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
> >>>>> confusing
> >>>>>>>>> about it
> >>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>> all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
> >>>>>>> confusing
> >>>>>>>>> to
> >>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>>>>>>>>> who
> >>>>>>>>>>>>>>>>>>>>>>>>>>> have been
> >>>>>>>>>>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> >>>>> Chung
> >>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> >>>>>>>>> tp@astronomer.io.inva
> >>>>>>>>>>>> lid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
> >>> the
> >>>>> term
> >>>>>>>> DAG.
> >>>>>>>>> The
> >>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>> problem
> >>>>>>>>>>>>>>>>>>>>>>>>>> (aside
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
> >>>>>>> myselfâto
> >>>>>>>>> stop
> >>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
> >>>>>>>>>>>>>>>>>>>>>>>>>> is to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
> >>> mentioned
> >>>>>>>>> âworkflowâ is
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> very
> >>>>>>>>>>>>>>>>>>>>>>>>>> accurate
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> >>>>>>> proposed,
> >>>>>>>>> but
> >>>>>>>>>>>> itâs a
> >>>>>>>>>>>>>>>>>>>>> bit
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
> >>> places
> >>>>> and
> >>>>>>> kept
> >>>>>>>>>>> finding
> >>>>>>>>>>>>>>>>>>>>> myself
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
> >>> wanting to
> >>>>>>>> clarify
> >>>>>>>>> âDAG
> >>>>>>>>>>>>>>>>>>>>>>> definitionâ
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
> >>> Scheffler
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> >>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
> >>> exactly
> >>>>> same
> >>>>>>>>>>>> observation,
> >>>>>>>>>>>>>>>>>>>>> had a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
> >>> always
> >>>>> an
> >>>>>>>>>>> introduction
> >>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
> >>>>>>>>>>>>>>>>>>>>>>>>>> joins
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
> >>> makes
> >>>>>>> sense.
> >>>>>>>>>>>>>>>>>>> Especially
> >>>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> >>>>>>> reasonable
> >>>>>>>>> step.
> >>>>>>>>>>>>>>>>>>>>> Concepts
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> >>>>> below
> >>>>>>> the
> >>>>>>>>> desk
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
> >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
> >>> keep
> >>>>>>>> details
> >>>>>>>>> of
> >>>>>>>>>>>>>>>>>>> python
> >>>>>>>>>>>>>>>>>>>>>> names
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
> >>> DAGâŠ
> >>>>>> but
> >>>>>>>> user
> >>>>>>>>>>>> facing
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
> >>> Hatter <
> >>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> >>>>>>>>> ryan.hatter@astronomer.io
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
> >>> swords... at
> >>>>>>> least
> >>>>>>>>> for
> >>>>>>>>>>>> now.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
> >>> meaning to
> >>>>>>>> Airflow
> >>>>>>>>>>>> users.
> >>>>>>>>>>>>>>>>>>>>>> Indeed,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
> >>>>>> mathematicians
> >>>>>>>> and
> >>>>>>>>>>>>>>>>>>> software
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
> >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> >>>>>> matter.
> >>>>>>>> For
> >>>>>>>>>>>> someone
> >>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
> >>>>> orchestration,
> >>>>>>> one
> >>>>>>>>> of
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
> >>> DAG?"
> >>>>>> The
> >>>>>>>>> answer
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> almost
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
> >>> graph.
> >>>>> You
> >>>>>>>> don't
> >>>>>>>>>>> need
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> worry
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
> >>>>> workflow."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> >>>>> least
> >>>>>> a
> >>>>>>>>> couple
> >>>>>>>>>>>>>>>>>>>>> important
> >>>>>>>>>>>>>>>>>>>>>>>>> reasons:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
> >>> mentioned
> >>>>>>> above,
> >>>>>>>>> "DAG"
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
> >>> confusing.
> >>>>>> We
> >>>>>>>> want
> >>>>>>>>>>>> Airflow
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
> >>> off the
> >>>>>> bat
> >>>>>>>>> creates
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> >>>>>>> Concepts*:
> >>>>>>>>> The
> >>>>>>>>>>> DAG
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
> >>> workflow. The
> >>>>>>>>> workflow
> >>>>>>>>>>>>>>>>>>> includes
> >>>>>>>>>>>>>>>>>>>>>> its
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> >>>>> other
> >>>>>>>>>>> parameters,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
> >>> Airflow
> >>>>>>>> homepage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> >>>>>>>>>>> https://airflow.apache.org/>>.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
> >>> created by
> >>>>>> the
> >>>>>>>>>>> community
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
> >>> workflows.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
> >>>>> Airflow?"
> >>>>>>> docs
> >>>>>>>>> page
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> >>>>>>>>>>>
> >>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
> >>> Airflow is
> >>>>>>>> without
> >>>>>>>>>>> using
> >>>>>>>>>>>>>>>>>>>>> "DAG."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
> >>> term
> >>>>> is
> >>>>>>>>>>> introduced
> >>>>>>>>>>>>>>>>>>> out
> >>>>>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
> >>> explain
> >>>>> it:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> >>>>> collection
> >>>>>>> of
> >>>>>>>>> tasks
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
> >>> in
> >>>>>> these
> >>>>>>>>>>>>>>>>>>> introductions
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> >>>>>> orchestrate
> >>>>>>>>> DAGs;
> >>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> orchestrates
> >>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> >>>>>> irrelevant
> >>>>>>>> to
> >>>>>>>>>>> almost
> >>>>>>>>>>>>>>>>>>>>> every
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
> >>> alias
> >>>>> for
> >>>>>>> the
> >>>>>>>>> term
> >>>>>>>>>>>> "DAG"
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
> >>> "DAG"
> >>>>>> with
> >>>>>>>>>>>> "workflow".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
> >>>>> away.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> ------------------------------------------------------------------
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> >>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> >>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> >>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
> >>>>>>>>>>> dev-help@airflow.apache.org>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Wei Lee <we...@gmail.com>.
For user-facing things (e.g., `from airflow import DAG`), Iâm more like -0. But for documentation, docstring, internal things and etc., we probably could still change most of them?
Best,
Wei
> On Feb 18, 2025, at 5:10âŻPM, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I also donât personally think itâs worth the pain (not to mention backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
>
> -ash
>
>> On 18 Feb 2025, at 04:40, Daniel Imberman <da...@gmail.com> wrote:
>>
>> I think my biggest concern is a marketing one and not a technical one.
>>
>> As has been mentioned on the thread the terms airflow and dag are kind of
>> synonymous and I certainly donât want to give the impression that we are
>> breaking more than we are breaking.
>>
>> I wouldnât die on this hill, but Iâm slightly concerned.
>>
>> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
>>
>>> Iâm not sure about adding ruff rules here đ€ I think ruff rules are best
>>> suited for user-facing things but not the airflow code base itself. If what
>>> we mean is adding a rule to avoid users using "DAG" *after* we rename it,
>>> it's definitely a +1000.
>>>
>>> I just created GitHub issues for this removing "DAG"
>>>
>>> https://github.com/apache/airflow/issues/46842
>>> https://github.com/apache/airflow/issues/46843
>>>
>>> One thing I'm not sure about is whether we want to get rid of `from
>>> airflow import DAG` as well. đ€
>>>
>>> Best,
>>> Wei
>>>
>>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
>>>> Hard to say until it's looked at :)
>>>>
>>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <ar...@gmail.com>
>>>> wrote:
>>>>
>>>>> I can take it up, it's mostly just a doc update right? Or are we doing
>>> code
>>>>> files replacement too?
>>>>> --
>>>>> Regards,
>>>>> Aritra Basu
>>>>>
>>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com> wrote:
>>>>>
>>>>>> Sounds like another +1000 files big PR is coming :) ? Any volunteers
>>> to
>>>>>> make it ? It's fun.
>>>>>>
>>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> +1 for ruff rules :)
>>>>>>>
>>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the
>>> Airflow
>>>>>>> docs, in line with the new UI changes and to make the renaming
>>>>>>> consistent across user-facing pages.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Omkar
>>>>>>>
>>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>>>>>>
>>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
>>>>>>>>
>>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
>>> wrote:
>>>>>>>>
>>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag"
>>>>> instead
>>>>>>> of
>>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the
>>>>> codebase
>>>>>>> with
>>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be
>>> đ€),
>>>>>>> should
>>>>>>>> we
>>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Wei
>>>>>>>>>
>>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
>>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
>>>>>>>>>> https://github.com/apache/airflow/pull/43325
>>>>>>>>>> Let me know if that's the direction we want to go.
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
>>>>>>>>>> <rb...@amazon.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
>>> synonymous
>>>>>> with
>>>>>>>>>>> workflows in Airflow through the way we talk and explain
>>> what
>>>>>>> Airflow
>>>>>>>>> is
>>>>>>>>>>> all about. Although folks would ask, I donât ever use the
>>>>>>>> mathematical
>>>>>>>>>>> definition of DAG.
>>>>>>>>>>> It will be challenging and possibly confusing for many
>>> users
>>>>>> making
>>>>>>>>> such a
>>>>>>>>>>> change - I would rather direct that energy to appending the
>>>>>> Oxford
>>>>>>>>>>> definition of "dag" to include a reference to workflows in
>>>>>> Airflow.
>>>>>>>> __
>>>>>>>>>>>
>>>>>>>>>>> -- Rajesh
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
>>>>>> <mailto:
>>>>>>>>>>> jarek@potiuk.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> CAUTION: This email originated from outside of the
>>>>> organization.
>>>>>> Do
>>>>>>>> not
>>>>>>>>>>> click links or open attachments unless you can confirm the
>>>>> sender
>>>>>>> and
>>>>>>>>> know
>>>>>>>>>>> the content is safe.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
>>>>> expéditeur
>>>>>>>>> externe.
>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
>>> si
>>>>> vous
>>>>>>> ne
>>>>>>>>> pouvez
>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
>>> pas
>>>>>>>> certain
>>>>>>>>> que
>>>>>>>>>>> le contenu ne présente aucun risque.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From Guido's post:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> "Around this time the renaming seems to have been renamed".
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Naming is hard.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> J.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
>>>>> droiddev510e@gmail.com
>>>>>>>>> <mailto:
>>>>>>>>>>> droiddev510e@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Yes, completely agree with above comments, dag is an
>>> Airflow
>>>>>> term
>>>>>>>> now
>>>>>>>>>>>> rather
>>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not,
>>> I've
>>>>>>>> worked
>>>>>>>>> with
>>>>>>>>>>>> work
>>>>>>>>>>>> folks who've been using dags in Airflow for years now
>>> and are
>>>>>> pro
>>>>>>>>> devs
>>>>>>>>>>> but
>>>>>>>>>>>> have trouble remembering the full form of a DAG! For
>>> them,
>>>>> dag
>>>>>> =
>>>>>>>>> Airflow.
>>>>>>>>>>>>
>>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
>>> will be
>>>>> a
>>>>>>>> major
>>>>>>>>>>>> behavior
>>>>>>>>>>>> change for the developer community, so we'll need a solid
>>>>> plan
>>>>>> on
>>>>>>>>> how to
>>>>>>>>>>>> introduce it, when we do.
>>>>>>>>>>>>
>>>>>>>>>>>> While we discuss this, I'd like to share about the Great
>>>>>> Renaming
>>>>>>>> in
>>>>>>>>> core
>>>>>>>>>>>> Python started by Guido back in 2009. We could probably
>>> get
>>>>>> some
>>>>>>>>>>> learnings
>>>>>>>>>>>> from there? Who knows!
>>>>>>>>>>>>
>>>>>>>>>>>> Guido's blog (2009):
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
>>>>>>>>>>> <
>>>>>>>>>
>>>>>>>
>>>>>
>>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
>>>>>>>>>
>>>>>>>>>>>> Follow-up discussion (2024):
>>>>>>>>>>>>
>>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082
>>>>>>> <
>>>>>>>>>>>
>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082>
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Omkar
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
>>> weilee.rx@gmail.com
>>>>>>>>> <mailto:
>>>>>>>>>>> weilee.rx@gmail.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I think we should probably just accept it as an airflow
>>>>> term.
>>>>>>> At
>>>>>>>>> least,
>>>>>>>>>>>>> thatâs how I understood it when I first used Airflow. I
>>>>> feel
>>>>>>>>> renaming
>>>>>>>>>>> it
>>>>>>>>>>>> at
>>>>>>>>>>>>> this stage would require considerable effort from
>>>>> maintainers
>>>>>>> and
>>>>>>>>>>>> existing
>>>>>>>>>>>>> users without providing equivalent benefits.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Wei
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
>>>>>> kaxilnaik@gmail.com
>>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
>>>>> this
>>>>>>>>> discussion
>>>>>>>>>>>> for
>>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
>>>>>>> disruption
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
>>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
>>>>>>>>> constance@astronomer.io.inva>lid>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
>>>>>> experience
>>>>>>>>> what a
>>>>>>>>>>>> dag
>>>>>>>>>>>>> is,
>>>>>>>>>>>>>>> they'll start talking about workflow attributes -
>>> stuff
>>>>>> like
>>>>>>>>> dags
>>>>>>>>>>>> being
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
>>>>>> doesn't
>>>>>>>>> come up.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
>>> embrace
>>>>> the
>>>>>>> name
>>>>>>>>> and
>>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
>>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
>>>>> vikram@astronomer.io.inva
>>>>>>>>> lid>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's an interesting discussion and I remember
>>>>> struggling
>>>>>>> with
>>>>>>>>> this
>>>>>>>>>>>>> when I
>>>>>>>>>>>>>>>> started working with Airflow.
>>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
>>>>>>>> established
>>>>>>>>>>>> concept
>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>> regardless of the origin.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am personally leaning towards the perspective
>>> best
>>>>>>>> expressed
>>>>>>>>> by
>>>>>>>>>>>>> Daniel
>>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
>>> than
>>>>>> the
>>>>>>>>> computer
>>>>>>>>>>>>>>> science
>>>>>>>>>>>>>>>> concept.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>> Vikram
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
>>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
>>>>>> onikolas@amazon.com.inva
>>>>>>>>> lid
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
>>> right
>>>>>> Ryan,
>>>>>>>> DAG
>>>>>>>>>>> isn't
>>>>>>>>>>>>>>> great
>>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
>>> cause
>>>>> much
>>>>>>>> more
>>>>>>>>>>>> wreckage
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>> it solves.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
>>>>> defining
>>>>>>>> DAG.
>>>>>>>>> I
>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>> we've been naturally doing that as a community
>>> for a
>>>>>> while
>>>>>>>> now
>>>>>>>>>>>> anyway,
>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>> that feels like a natural step.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Niko
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
>>>>>>>>> ash@apache.org>>
>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
>>>>>> dev@airflow.apache.org
>>>>>>>>
>>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
>>> term
>>>>>> "DAG"
>>>>>>>>> for end
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
>>>>>>>>> organization.
>>>>>>>>>>> Do
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> click links or open attachments unless you can
>>> confirm
>>>>>> the
>>>>>>>>> sender
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>>> the content is safe.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
>>> dâun
>>>>>>>>> expéditeur
>>>>>>>>>>>>>>> externe.
>>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
>>>>>> jointe
>>>>>>> si
>>>>>>>>> vous
>>>>>>>>>>> ne
>>>>>>>>>>>>>>>> pouvez
>>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
>>> vous
>>>>>> nâĂȘtes
>>>>>>>> pas
>>>>>>>>>>>> certain
>>>>>>>>>>>>>>>> que
>>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
>>> term â
>>>>>>>> getting
>>>>>>>>> to
>>>>>>>>>>>>> re-use
>>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
>>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
>>>>> sides
>>>>>>> have
>>>>>>>>> good
>>>>>>>>>>>>>>> reasons
>>>>>>>>>>>>>>>>> presented.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
>>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
>>> from,
>>>>>>> "well,
>>>>>>>>> no
>>>>>>>>>>> one
>>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>> knows but..." and then make something up.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
>>>>>>>>> jarek@potiuk.com
>>>>>>>>>>> <ma...@potiuk.com>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
>>> the
>>>>>>>>>>> tongue-twister,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
>>>>>>>>> jarek@potiuk.com
>>>>>>>>>>> <ma...@potiuk.com>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
>>> would
>>>>> very
>>>>>>>> much
>>>>>>>>> want
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
>>> further.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
>>> it's a
>>>>>>>>>>>> tongue-twister,
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
>>> pronounce
>>>>>> it
>>>>>>>>>>> properly.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
>>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
>>>>>>>> brent@astronomer.io.inva
>>>>>>>>>> lid>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
>>>>> starting
>>>>>> on
>>>>>>>>> AIP-38
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> modernize
>>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
>>>>>>>>> descriptive of
>>>>>>>>>>>> what
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
>>> is an
>>>>>>>>>>> implementation
>>>>>>>>>>>>>>>>> detail.
>>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
>>> point
>>>>>> "DAG"
>>>>>>>> has
>>>>>>>>>>> become
>>>>>>>>>>>>>>>> "dag"
>>>>>>>>>>>>>>>>>>> , a
>>>>>>>>>>>>>>>>>>>>> word in its own right.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
>>>>>> discussion,
>>>>>>>>> Airflow
>>>>>>>>>>>>>>>> Summit
>>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
>>>>>> embrace
>>>>>>>>> "dag".
>>>>>>>>>>> A
>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
>>>>> concept
>>>>>>>>> behind
>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> word. I
>>>>>>>>>>>>>>>>>>>>> think that is much less effort than
>>> refactoring so
>>>>>>> much
>>>>>>>>> code,
>>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
>>>>> questions,
>>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
>>> Standish
>>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
>>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
>>> of
>>>>>> pain
>>>>>>>> for
>>>>>>>>>>>>>>>>> questionable
>>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
>>> idea. I
>>>>>>> agree
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> association
>>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
>>>>>>> invasive
>>>>>>>>>>>>>>> mitigations.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
>>>>>> de-emphasize
>>>>>>>> the
>>>>>>>>>>> math
>>>>>>>>>>>>>>> nerd
>>>>>>>>>>>>>>>>>>>>> origins
>>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
>>> website /
>>>>>> etc,
>>>>>>>>> *never
>>>>>>>>>>>>>>>> define*
>>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
>>>>>> graph.
>>>>>>>>> Always
>>>>>>>>>>>>>>> define
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
>>> like a
>>>>>>>>> historical
>>>>>>>>>>>>>>>>> footnote,
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
>>> hidden.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
>>> docs
>>>>> in
>>>>>>>>> general
>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>>> writing
>>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
>>> "DAGs"
>>>>>> etc.
>>>>>>>> The
>>>>>>>>>>>> upper
>>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
>>> "dag" in
>>>>>>>>> airlfow is
>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
>>> "DAGs"
>>>>> is
>>>>>>> not
>>>>>>>>> really
>>>>>>>>>>>>>>>>>>> unhelpful.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
>>> is
>>>>> its
>>>>>>> own
>>>>>>>>>>> thing,
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> *not* strictly
>>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
>>> nobody
>>>>>> knows
>>>>>>>>> about
>>>>>>>>>>>>>>>> anyway),
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
>>> normal
>>>>>>> people
>>>>>>>>>>>>>>> understand.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
>>> <
>>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
>>> will
>>>>> be
>>>>>>>>> extremely
>>>>>>>>>>>>>>>>>>>>> difficult to
>>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
>>> will
>>>>>> make
>>>>>>> a
>>>>>>>>> lot of
>>>>>>>>>>>>>>>>>>> "google"
>>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
>>>>> finding
>>>>>>> the
>>>>>>>>> right
>>>>>>>>>>>>>>>>>>> answers.
>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
>>> besides the
>>>>>>>>> community
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
>>>>> examples,
>>>>>>>>> problems
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> solutions
>>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
>>> remember
>>>>>> that
>>>>>>>>> all the
>>>>>>>>>>>> AI
>>>>>>>>>>>>>>>>>>>>> trained
>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
>>> matching
>>>>>>> queries
>>>>>>>> of
>>>>>>>>>>>> people.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
>>>>>> switch.
>>>>>>>>> And if
>>>>>>>>>>> we
>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
>>> instead
>>>>> of
>>>>>>>>> `dag` if
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> above
>>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
>>> that it
>>>>>>> might
>>>>>>>>> cause
>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>> problems
>>>>>>>>>>>>>>>>>>>>>>> than it solves.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
>>> think
>>>>>>> it's
>>>>>>>> a
>>>>>>>>> good
>>>>>>>>>>>>>>>> idea, I
>>>>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>>> with it.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> J,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
>>> Bhakat
>>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
>>> <mailto:
>>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
>>> sense
>>>>> to
>>>>>>>>> someone
>>>>>>>>>>> new
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> workflow
>>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
>>> nature
>>>>>> of
>>>>>>>>> being
>>>>>>>>>>>>>>> acyclic.
>>>>>>>>>>>>>>>>>>>>> Sure,
>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
>>> workaround it.
>>>>>>>> Still,
>>>>>>>>> in
>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
>>>>>>> workflow
>>>>>>>>>>>>>>>>>>> orchestration.
>>>>>>>>>>>>>>>>>>>>>> Most
>>>>>>>>>>>>>>>>>>>>>>>> (*if
>>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
>>>>> covered
>>>>>>>>> using an
>>>>>>>>>>>>>>>> acyclic
>>>>>>>>>>>>>>>>>>>>>> manner
>>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
>>> idempotency. So
>>>>> I
>>>>>>>> would
>>>>>>>>> want
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> "acyclic"
>>>>>>>>>>>>>>>>>>>>>>>> word to stick.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Avi
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
>>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
>>> bernd.stroehle@kosakya.de>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
>>>>> Airflow
>>>>>>> Fan;
>>>>>>>>> so
>>>>>>>>>>> many
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>> ideas.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
>>>>>> replaced
>>>>>>> by
>>>>>>>>> the
>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>> general
>>>>>>>>>>>>>>>>>>>>>>>> Term
>>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
>>>>>> (Petri)
>>>>>>>> Net
>>>>>>>>>>> (AFN)
>>>>>>>>>>>>>>>>>>> (maybe
>>>>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>>> a direction);
>>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
>>>>> Graph
>>>>>>>>> Database.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
>>> Graph
>>>>>> (AFG)
>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>> assigned
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
>>> of a
>>>>>>>> library.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
>>>>> structure
>>>>>>>> than
>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>>>>> Graph.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
>>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
>>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
>>>>>>>>>>> bernd.stroehle@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
>>> <ikholopov@google.com.INVA
>>>>>>>>> <mailto:
>>>>>>>>>>> ikholopov@google.com.INVA>LID>
>>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
>>>>>>>>> dev@airflow.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
>>> term
>>>>>>> "DAG"
>>>>>>>>> for
>>>>>>>>>>> end
>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
>>>>>> suboptimal,
>>>>>>> it
>>>>>>>>> is
>>>>>>>>>>> part
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
>>> levels,
>>>>> that
>>>>>>> any
>>>>>>>>>>> attempt
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
>>> reduce it.
>>>>>> The
>>>>>>>>> only
>>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>>> worse
>>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
>>> when
>>>>>>> there
>>>>>>>>> are
>>>>>>>>>>> two
>>>>>>>>>>>>>>> ways
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> define
>>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
>>>>> tutorials
>>>>>>>> will
>>>>>>>>>>>> suddenly
>>>>>>>>>>>>>>>>>>>>> become
>>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
>>> as
>>>>>>> "DAG"s.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
>>> users
>>>>> away
>>>>>>>> with
>>>>>>>>> a
>>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
>>>>>> breaking
>>>>>>>>> changes
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
>>> are
>>>>>>>> looking
>>>>>>>>> for.
>>>>>>>>>>>>>>>>>>>>> Attempting
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
>>>>> interpreted
>>>>>>> as
>>>>>>>> an
>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>> stronger
>>>>>>>>>>>>>>>>>>>>>>>> signal
>>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
>>> the
>>>>>> term
>>>>>>>> in
>>>>>>>>> the
>>>>>>>>>>>> long
>>>>>>>>>>>>>>>>>>>>> run. I
>>>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
>>> it in
>>>>>> the
>>>>>>>> DAG
>>>>>>>>>>>>>>> definition
>>>>>>>>>>>>>>>>>>>>>>>> interface.
>>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
>>> other
>>>>>> places
>>>>>>>> in
>>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Igor
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
>>> Harenslak
>>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
>>>>>>>> bas@astronomer.io.inva
>>>>>>>>>> lid
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
>>> have
>>>>>>> already
>>>>>>>>> faded
>>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
>>>>>> create
>>>>>>>>> cyclic
>>>>>>>>>>>>>>>>>>> graphs,
>>>>>>>>>>>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
>>>>> these
>>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>>>> vanishing
>>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
>>>>>>> perspective I
>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
>>>>>> naming:
>>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
>>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
>>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
>>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
>>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
>>> name.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
>>> change, I
>>>>>>> suggest
>>>>>>>>>>>> defining
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> clear
>>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
>>> the
>>>>>>>>> deprecation in
>>>>>>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>> 3,
>>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Bas
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
>>>>>>>> neil4real@gmail.com
>>>>>>>>>>> <ma...@gmail.com>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
>>>>>>>> especially
>>>>>>>>> when
>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
>>> wholeheartedly.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
>>>>> confusing
>>>>>>>>> about it
>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>> all,
>>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
>>>>>>> confusing
>>>>>>>>> to
>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>> who
>>>>>>>>>>>>>>>>>>>>>>>>>>> have been
>>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
>>>>> Chung
>>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
>>>>>>>>> tp@astronomer.io.inva
>>>>>>>>>>>> lid
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
>>> the
>>>>> term
>>>>>>>> DAG.
>>>>>>>>> The
>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>>>>>>>> (aside
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
>>>>>>> myselfâto
>>>>>>>>> stop
>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
>>>>>>>>>>>>>>>>>>>>>>>>>> is to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
>>> mentioned
>>>>>>>>> âworkflowâ is
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>>> accurate
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
>>>>>>> proposed,
>>>>>>>>> but
>>>>>>>>>>>> itâs a
>>>>>>>>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
>>> places
>>>>> and
>>>>>>> kept
>>>>>>>>>>> finding
>>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
>>> wanting to
>>>>>>>> clarify
>>>>>>>>> âDAG
>>>>>>>>>>>>>>>>>>>>>>> definitionâ
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
>>> Scheffler
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
>>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
>>> exactly
>>>>> same
>>>>>>>>>>>> observation,
>>>>>>>>>>>>>>>>>>>>> had a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
>>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
>>> always
>>>>> an
>>>>>>>>>>> introduction
>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
>>>>>>>>>>>>>>>>>>>>>>>>>> joins
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
>>> makes
>>>>>>> sense.
>>>>>>>>>>>>>>>>>>> Especially
>>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
>>>>>>> reasonable
>>>>>>>>> step.
>>>>>>>>>>>>>>>>>>>>> Concepts
>>>>>>>>>>>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
>>>>> below
>>>>>>> the
>>>>>>>>> desk
>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
>>> keep
>>>>>>>> details
>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> python
>>>>>>>>>>>>>>>>>>>>>> names
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
>>> DAGâŠ
>>>>>> but
>>>>>>>> user
>>>>>>>>>>>> facing
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
>>> Hatter <
>>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
>>>>>>>>> ryan.hatter@astronomer.io
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
>>> swords... at
>>>>>>> least
>>>>>>>>> for
>>>>>>>>>>>> now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
>>> meaning to
>>>>>>>> Airflow
>>>>>>>>>>>> users.
>>>>>>>>>>>>>>>>>>>>>> Indeed,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
>>>>>> mathematicians
>>>>>>>> and
>>>>>>>>>>>>>>>>>>> software
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
>>>>>> matter.
>>>>>>>> For
>>>>>>>>>>>> someone
>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
>>>>> orchestration,
>>>>>>> one
>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
>>> DAG?"
>>>>>> The
>>>>>>>>> answer
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> almost
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
>>> graph.
>>>>> You
>>>>>>>> don't
>>>>>>>>>>> need
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> worry
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
>>>>> workflow."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
>>>>> least
>>>>>> a
>>>>>>>>> couple
>>>>>>>>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
>>> mentioned
>>>>>>> above,
>>>>>>>>> "DAG"
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
>>> confusing.
>>>>>> We
>>>>>>>> want
>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
>>> off the
>>>>>> bat
>>>>>>>>> creates
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
>>>>>>> Concepts*:
>>>>>>>>> The
>>>>>>>>>>> DAG
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
>>> workflow. The
>>>>>>>>> workflow
>>>>>>>>>>>>>>>>>>> includes
>>>>>>>>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
>>>>> other
>>>>>>>>>>> parameters,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
>>> Airflow
>>>>>>>> homepage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
>>>>>>>>>>> https://airflow.apache.org/>>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
>>> created by
>>>>>> the
>>>>>>>>>>> community
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
>>> workflows.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
>>>>> Airflow?"
>>>>>>> docs
>>>>>>>>> page
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
>>>>>>>>>>>
>>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html
>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
>>> Airflow is
>>>>>>>> without
>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>> "DAG."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
>>> term
>>>>> is
>>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
>>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
>>> explain
>>>>> it:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
>>>>> collection
>>>>>>> of
>>>>>>>>> tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
>>> in
>>>>>> these
>>>>>>>>>>>>>>>>>>> introductions
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
>>>>>> orchestrate
>>>>>>>>> DAGs;
>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> orchestrates
>>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
>>>>>> irrelevant
>>>>>>>> to
>>>>>>>>>>> almost
>>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
>>> alias
>>>>> for
>>>>>>> the
>>>>>>>>> term
>>>>>>>>>>>> "DAG"
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
>>> "DAG"
>>>>>> with
>>>>>>>>>>>> "workflow".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
>>>>> away.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> ------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
>>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
>>>>>>>>>>> dev-help@airflow.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Airflow should deprecate the term "DAG" for end users
Posted by Ash Berlin-Taylor <as...@apache.org>.
I also donât personally think itâs worth the pain (not to mention backcompat workaround) to rename DAG to Dag, so Iâd be -0.5 on that.
-ash
> On 18 Feb 2025, at 04:40, Daniel Imberman <da...@gmail.com> wrote:
>
> I think my biggest concern is a marketing one and not a technical one.
>
> As has been mentioned on the thread the terms airflow and dag are kind of
> synonymous and I certainly donât want to give the impression that we are
> breaking more than we are breaking.
>
> I wouldnât die on this hill, but Iâm slightly concerned.
>
> On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
>
>> Iâm not sure about adding ruff rules here đ€ I think ruff rules are best
>> suited for user-facing things but not the airflow code base itself. If what
>> we mean is adding a rule to avoid users using "DAG" *after* we rename it,
>> it's definitely a +1000.
>>
>> I just created GitHub issues for this removing "DAG"
>>
>> https://github.com/apache/airflow/issues/46842
>> https://github.com/apache/airflow/issues/46843
>>
>> One thing I'm not sure about is whether we want to get rid of `from
>> airflow import DAG` as well. đ€
>>
>> Best,
>> Wei
>>
>> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
>>> Hard to say until it's looked at :)
>>>
>>> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <ar...@gmail.com>
>>> wrote:
>>>
>>>> I can take it up, it's mostly just a doc update right? Or are we doing
>> code
>>>> files replacement too?
>>>> --
>>>> Regards,
>>>> Aritra Basu
>>>>
>>>> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com> wrote:
>>>>
>>>>> Sounds like another +1000 files big PR is coming :) ? Any volunteers
>> to
>>>>> make it ? It's fun.
>>>>>
>>>>> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com>
>> wrote:
>>>>>
>>>>>> +1 for ruff rules :)
>>>>>>
>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in the
>> Airflow
>>>>>> docs, in line with the new UI changes and to make the renaming
>>>>>> consistent across user-facing pages.
>>>>>>
>>>>>> Regards,
>>>>>> Omkar
>>>>>>
>>>>>> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>>>>>>
>>>>>>> +1 :) . Maybe we could add a ruff rule for that :)
>>>>>>>
>>>>>>> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
>> wrote:
>>>>>>>
>>>>>>>> It seems that our current conclusion is to use "dag" or "Dag"
>>>> instead
>>>>>> of
>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in the
>>>> codebase
>>>>>> with
>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might be
>> đ€),
>>>>>> should
>>>>>>> we
>>>>>>>> at least avoid introducing new "DAG" in the following PRs?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Wei
>>>>>>>>
>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI:
>>>>>>>>> https://github.com/apache/airflow/pull/43325
>>>>>>>>> Let me know if that's the direction we want to go.
>>>>>>>>>
>>>>>>>>> On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
>>>>>>>>> <rb...@amazon.com.invalid> wrote:
>>>>>>>>>
>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is
>> synonymous
>>>>> with
>>>>>>>>>> workflows in Airflow through the way we talk and explain
>> what
>>>>>> Airflow
>>>>>>>> is
>>>>>>>>>> all about. Although folks would ask, I donât ever use the
>>>>>>> mathematical
>>>>>>>>>> definition of DAG.
>>>>>>>>>> It will be challenging and possibly confusing for many
>> users
>>>>> making
>>>>>>>> such a
>>>>>>>>>> change - I would rather direct that energy to appending the
>>>>> Oxford
>>>>>>>>>> definition of "dag" to include a reference to workflows in
>>>>> Airflow.
>>>>>>> __
>>>>>>>>>>
>>>>>>>>>> -- Rajesh
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
>>>>> <mailto:
>>>>>>>>>> jarek@potiuk.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> CAUTION: This email originated from outside of the
>>>> organization.
>>>>> Do
>>>>>>> not
>>>>>>>>>> click links or open attachments unless you can confirm the
>>>> sender
>>>>>> and
>>>>>>>> know
>>>>>>>>>> the content is safe.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
>>>> expéditeur
>>>>>>>> externe.
>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
>> si
>>>> vous
>>>>>> ne
>>>>>>>> pouvez
>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
>> pas
>>>>>>> certain
>>>>>>>> que
>>>>>>>>>> le contenu ne présente aucun risque.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From Guido's post:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> "Around this time the renaming seems to have been renamed".
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Naming is hard.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> J.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
>>>> droiddev510e@gmail.com
>>>>>>>> <mailto:
>>>>>>>>>> droiddev510e@gmail.com>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Yes, completely agree with above comments, dag is an
>> Airflow
>>>>> term
>>>>>>> now
>>>>>>>>>>> rather
>>>>>>>>>>> than just a "directed acyclic graph". Believe it or not,
>> I've
>>>>>>> worked
>>>>>>>> with
>>>>>>>>>>> work
>>>>>>>>>>> folks who've been using dags in Airflow for years now
>> and are
>>>>> pro
>>>>>>>> devs
>>>>>>>>>> but
>>>>>>>>>>> have trouble remembering the full form of a DAG! For
>> them,
>>>> dag
>>>>> =
>>>>>>>> Airflow.
>>>>>>>>>>>
>>>>>>>>>>> workflow, pipeline, flow are surely better terms but
>> will be
>>>> a
>>>>>>> major
>>>>>>>>>>> behavior
>>>>>>>>>>> change for the developer community, so we'll need a solid
>>>> plan
>>>>> on
>>>>>>>> how to
>>>>>>>>>>> introduce it, when we do.
>>>>>>>>>>>
>>>>>>>>>>> While we discuss this, I'd like to share about the Great
>>>>> Renaming
>>>>>>> in
>>>>>>>> core
>>>>>>>>>>> Python started by Guido back in 2009. We could probably
>> get
>>>>> some
>>>>>>>>>> learnings
>>>>>>>>>>> from there? Who knows!
>>>>>>>>>>>
>>>>>>>>>>> Guido's blog (2009):
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
>>>>>>>>>> <
>>>>>>>>
>>>>>>
>>>>
>> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
>>>>>>>>
>>>>>>>>>>> Follow-up discussion (2024):
>>>>>>>>>>>
>>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082
>>>>>> <
>>>>>>>>>>
>>>> https://discuss.python.org/t/finishing-the-great-renaming/54082>
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Omkar
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
>> weilee.rx@gmail.com
>>>>>>>> <mailto:
>>>>>>>>>> weilee.rx@gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I think we should probably just accept it as an airflow
>>>> term.
>>>>>> At
>>>>>>>> least,
>>>>>>>>>>>> thatâs how I understood it when I first used Airflow. I
>>>> feel
>>>>>>>> renaming
>>>>>>>>>> it
>>>>>>>>>>> at
>>>>>>>>>>>> this stage would require considerable effort from
>>>> maintainers
>>>>>> and
>>>>>>>>>>> existing
>>>>>>>>>>>> users without providing equivalent benefits.
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Wei
>>>>>>>>>>>>
>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
>>>>> kaxilnaik@gmail.com
>>>>>>>>>> <ma...@gmail.com>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle
>>>> this
>>>>>>>> discussion
>>>>>>>>>>> for
>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much
>>>>>> disruption
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau
>>>>>>>>>>>>> <constance@astronomer.io.inva <mailto:
>>>>>>>> constance@astronomer.io.inva>lid>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> In my experience, when you ask those with Airflow
>>>>> experience
>>>>>>>> what a
>>>>>>>>>>> dag
>>>>>>>>>>>> is,
>>>>>>>>>>>>>> they'll start talking about workflow attributes -
>> stuff
>>>>> like
>>>>>>>> dags
>>>>>>>>>>> being
>>>>>>>>>>>> a
>>>>>>>>>>>>>> series of steps or tasks with owners. The structure
>>>>> doesn't
>>>>>>>> come up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to
>> embrace
>>>> the
>>>>>> name
>>>>>>>> and
>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
>>>>>>>>>>>> <vikram@astronomer.io.inva <mailto:
>>>> vikram@astronomer.io.inva
>>>>>>>> lid>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's an interesting discussion and I remember
>>>> struggling
>>>>>> with
>>>>>>>> this
>>>>>>>>>>>> when I
>>>>>>>>>>>>>>> started working with Airflow.
>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an
>>>>>>> established
>>>>>>>>>>> concept
>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>> regardless of the origin.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am personally leaning towards the perspective
>> best
>>>>>>> expressed
>>>>>>>> by
>>>>>>>>>>>> Daniel
>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather
>> than
>>>>> the
>>>>>>>> computer
>>>>>>>>>>>>>> science
>>>>>>>>>>>>>>> concept.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Vikram
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
>>>>>>>>>>>>>> <onikolas@amazon.com.inva <mailto:
>>>>> onikolas@amazon.com.inva
>>>>>>>> lid
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're
>> right
>>>>> Ryan,
>>>>>>> DAG
>>>>>>>>>> isn't
>>>>>>>>>>>>>> great
>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will
>> cause
>>>> much
>>>>>>> more
>>>>>>>>>>> wreckage
>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>> it solves.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also agree with the idea to just move away from
>>>> defining
>>>>>>> DAG.
>>>>>>>> I
>>>>>>>>>>> think
>>>>>>>>>>>>>>>> we've been naturally doing that as a community
>> for a
>>>>> while
>>>>>>> now
>>>>>>>>>>> anyway,
>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>> that feels like a natural step.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Niko
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
>>>>>>>> ash@apache.org>>
>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
>>>>> dev@airflow.apache.org
>>>>>>>
>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the
>> term
>>>>> "DAG"
>>>>>>>> for end
>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the
>>>>>>>> organization.
>>>>>>>>>> Do
>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> click links or open attachments unless you can
>> confirm
>>>>> the
>>>>>>>> sender
>>>>>>>>>>> and
>>>>>>>>>>>>>>> know
>>>>>>>>>>>>>>>> the content is safe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient
>> dâun
>>>>>>>> expéditeur
>>>>>>>>>>>>>> externe.
>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
>>>>> jointe
>>>>>> si
>>>>>>>> vous
>>>>>>>>>> ne
>>>>>>>>>>>>>>> pouvez
>>>>>>>>>>>>>>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
>> vous
>>>>> nâĂȘtes
>>>>>>> pas
>>>>>>>>>>> certain
>>>>>>>>>>>>>>> que
>>>>>>>>>>>>>>>> le contenu ne présente aucun risque.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best argument in favour of keeping âdagsâ as a
>> term â
>>>>>>> getting
>>>>>>>> to
>>>>>>>>>>>> re-use
>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg <
>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In all seriousness: I donât mind either way, both
>>>> sides
>>>>>> have
>>>>>>>> good
>>>>>>>>>>>>>> reasons
>>>>>>>>>>>>>>>> presented.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish
>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.INVA <mailto:
>>>>>>>>>> daniel.standish@astronomer.io.INVA>LID> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes
>> from,
>>>>>> "well,
>>>>>>>> no
>>>>>>>>>> one
>>>>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>> knows but..." and then make something up.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
>>>>>>>> jarek@potiuk.com
>>>>>>>>>> <ma...@potiuk.com>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is
>> the
>>>>>>>>>> tongue-twister,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
>>>>>>>> jarek@potiuk.com
>>>>>>>>>> <ma...@potiuk.com>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I
>> would
>>>> very
>>>>>>> much
>>>>>>>> want
>>>>>>>>>>> to
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> able
>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it
>> further.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk
>> it's a
>>>>>>>>>>> tongue-twister,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to
>> pronounce
>>>>> it
>>>>>>>>>> properly.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> J.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
>>>>>>>>>>>>>>>>>> <brent@astronomer.io.inva <mailto:
>>>>>>> brent@astronomer.io.inva
>>>>>>>>> lid>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when
>>>> starting
>>>>> on
>>>>>>>> AIP-38
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> modernize
>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more
>>>>>>>> descriptive of
>>>>>>>>>>> what
>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph
>> is an
>>>>>>>>>> implementation
>>>>>>>>>>>>>>>> detail.
>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this
>> point
>>>>> "DAG"
>>>>>>> has
>>>>>>>>>> become
>>>>>>>>>>>>>>> "dag"
>>>>>>>>>>>>>>>>>> , a
>>>>>>>>>>>>>>>>>>>> word in its own right.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community
>>>>> discussion,
>>>>>>>> Airflow
>>>>>>>>>>>>>>> Summit
>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's
>>>>> embrace
>>>>>>>> "dag".
>>>>>>>>>> A
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical
>>>> concept
>>>>>>>> behind
>>>>>>>>>> that
>>>>>>>>>>>>>>>> word. I
>>>>>>>>>>>>>>>>>>>> think that is much less effort than
>> refactoring so
>>>>>> much
>>>>>>>> code,
>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow
>>>> questions,
>>>>>>> etc.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
>> Standish
>>>>>>>>>>>>>>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
>>>>>>>>>> daniel.standish@astronomer.io.inva>lid> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot
>> of
>>>>> pain
>>>>>>> for
>>>>>>>>>>>>>>>> questionable
>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the
>> idea. I
>>>>>> agree
>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> association
>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less
>>>>>> invasive
>>>>>>>>>>>>>> mitigations.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to
>>>>> de-emphasize
>>>>>>> the
>>>>>>>>>> math
>>>>>>>>>>>>>> nerd
>>>>>>>>>>>>>>>>>>>> origins
>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs /
>> website /
>>>>> etc,
>>>>>>>> *never
>>>>>>>>>>>>>>> define*
>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic
>>>>> graph.
>>>>>>>> Always
>>>>>>>>>>>>>> define
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is
>> like a
>>>>>>>> historical
>>>>>>>>>>>>>>>> footnote,
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere
>> hidden.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the
>> docs
>>>> in
>>>>>>>> general
>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>> writing
>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" /
>> "DAGs"
>>>>> etc.
>>>>>>> The
>>>>>>>>>>> upper
>>>>>>>>>>>>>>> case
>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but
>> "dag" in
>>>>>>>> airlfow is
>>>>>>>>>>> just
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with
>> "DAGs"
>>>> is
>>>>>> not
>>>>>>>> really
>>>>>>>>>>>>>>>>>> unhelpful.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow
>> is
>>>> its
>>>>>> own
>>>>>>>>>> thing,
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> *not* strictly
>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which
>> nobody
>>>>> knows
>>>>>>>> about
>>>>>>>>>>>>>>> anyway),
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that
>> normal
>>>>>> people
>>>>>>>>>>>>>> understand.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
>> <
>>>>>>>>>> jarek@potiuk.com <ma...@potiuk.com>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it
>> will
>>>> be
>>>>>>>> extremely
>>>>>>>>>>>>>>>>>>>> difficult to
>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it
>> will
>>>>> make
>>>>>> a
>>>>>>>> lot of
>>>>>>>>>>>>>>>>>> "google"
>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not
>>>> finding
>>>>>> the
>>>>>>>> right
>>>>>>>>>>>>>>>>>> answers.
>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow -
>> besides the
>>>>>>>> community
>>>>>>>>>> and
>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of
>>>> examples,
>>>>>>>> problems
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> solutions
>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to
>> remember
>>>>> that
>>>>>>>> all the
>>>>>>>>>>> AI
>>>>>>>>>>>>>>>>>>>> trained
>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly
>> matching
>>>>>> queries
>>>>>>> of
>>>>>>>>>>> people.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily
>>>>> switch.
>>>>>>>> And if
>>>>>>>>>> we
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline
>> instead
>>>> of
>>>>>>>> `dag` if
>>>>>>>>>>> not
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> above
>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor
>> that it
>>>>>> might
>>>>>>>> cause
>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>> problems
>>>>>>>>>>>>>>>>>>>>>> than it solves.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will
>> think
>>>>>> it's
>>>>>>> a
>>>>>>>> good
>>>>>>>>>>>>>>> idea, I
>>>>>>>>>>>>>>>>>>>> am
>>>>>>>>>>>>>>>>>>>>> ok
>>>>>>>>>>>>>>>>>>>>>> with it.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> J,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
>> Bhakat
>>>>>>>>>>>>>>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
>> <mailto:
>>>>>>>>>> abhishek.bhakat@astronomer.io.inva>lid> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less
>> sense
>>>> to
>>>>>>>> someone
>>>>>>>>>> new
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> workflow
>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the
>> nature
>>>>> of
>>>>>>>> being
>>>>>>>>>>>>>> acyclic.
>>>>>>>>>>>>>>>>>>>> Sure,
>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to
>> workaround it.
>>>>>>> Still,
>>>>>>>> in
>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>> opinion,
>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in
>>>>>> workflow
>>>>>>>>>>>>>>>>>> orchestration.
>>>>>>>>>>>>>>>>>>>>> Most
>>>>>>>>>>>>>>>>>>>>>>> (*if
>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be
>>>> covered
>>>>>>>> using an
>>>>>>>>>>>>>>> acyclic
>>>>>>>>>>>>>>>>>>>>> manner
>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the
>> idempotency. So
>>>> I
>>>>>>> would
>>>>>>>> want
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> "acyclic"
>>>>>>>>>>>>>>>>>>>>>>> word to stick.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>>>> Avi
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
>>>>>>>>>> bernd.stroehle@kosakya.de <mailto:
>> bernd.stroehle@kosakya.de>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an
>>>> Airflow
>>>>>> Fan;
>>>>>>>> so
>>>>>>>>>> many
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> ideas.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be
>>>>> replaced
>>>>>> by
>>>>>>>> the
>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>> general
>>>>>>>>>>>>>>>>>>>>>>> Term
>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
>>>>> (Petri)
>>>>>>> Net
>>>>>>>>>> (AFN)
>>>>>>>>>>>>>>>>>> (maybe
>>>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>> a direction);
>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a
>>>> Graph
>>>>>>>> Database.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
>> Graph
>>>>> (AFG)
>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>> assigned
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member
>> of a
>>>>>>> library.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different
>>>> structure
>>>>>>> than
>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> configuration
>>>>>>>>>>>>>>>>>>>>>>>> Graph.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle
>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916
>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
>>>>>>>>>> bernd.stroehle@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov
>> <ikholopov@google.com.INVA
>>>>>>>> <mailto:
>>>>>>>>>> ikholopov@google.com.INVA>LID>
>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
>>>>>>>> dev@airflow.apache.org
>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the
>> term
>>>>>> "DAG"
>>>>>>>> for
>>>>>>>>>> end
>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly
>>>>> suboptimal,
>>>>>> it
>>>>>>>> is
>>>>>>>>>> part
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many
>> levels,
>>>> that
>>>>>> any
>>>>>>>>>> attempt
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not
>> reduce it.
>>>>> The
>>>>>>>> only
>>>>>>>>>>> thing
>>>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>>>>>>> worse
>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is
>> when
>>>>>> there
>>>>>>>> are
>>>>>>>>>> two
>>>>>>>>>>>>>> ways
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> define
>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and
>>>> tutorials
>>>>>>> will
>>>>>>>>>>> suddenly
>>>>>>>>>>>>>>>>>>>> become
>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows
>> as
>>>>>> "DAG"s.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the
>> users
>>>> away
>>>>>>> with
>>>>>>>> a
>>>>>>>>>>> number
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>> breaking
>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more
>>>>> breaking
>>>>>>>> changes
>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people
>> are
>>>>>>> looking
>>>>>>>> for.
>>>>>>>>>>>>>>>>>>>> Attempting
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be
>>>> interpreted
>>>>>> as
>>>>>>> an
>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>> stronger
>>>>>>>>>>>>>>>>>>>>>>> signal
>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing
>> the
>>>>> term
>>>>>>> in
>>>>>>>> the
>>>>>>>>>>> long
>>>>>>>>>>>>>>>>>>>> run. I
>>>>>>>>>>>>>>>>>>>>>>> even
>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating
>> it in
>>>>> the
>>>>>>> DAG
>>>>>>>>>>>>>> definition
>>>>>>>>>>>>>>>>>>>>>>> interface.
>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in
>> other
>>>>> places
>>>>>>> in
>>>>>>>>>>> Airflow,
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>> there are plenty.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>>>>>>>>>>>>> Igor
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
>> Harenslak
>>>>>>>>>>>>>>>>>>>>>> <bas@astronomer.io.inva <mailto:
>>>>>>> bas@astronomer.io.inva
>>>>>>>>> lid
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
>> have
>>>>>> already
>>>>>>>> faded
>>>>>>>>>>> over
>>>>>>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to
>>>>> create
>>>>>>>> cyclic
>>>>>>>>>>>>>>>>>> graphs,
>>>>>>>>>>>>>>>>>>>>> e.g.
>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine
>>>> these
>>>>>>>>>> properties
>>>>>>>>>>>>>>>>>>>>> vanishing
>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that
>>>>>> perspective I
>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>> changing
>>>>>>>>>>>>>>>>>>>>>>>>> âDAG" to a more generic name.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do
>>>>> naming:
>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline
>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow
>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow
>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow
>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think âworkflowâ is the most fitting
>> name.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this
>> change, I
>>>>>> suggest
>>>>>>>>>>> defining
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> clear
>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce
>> the
>>>>>>>> deprecation in
>>>>>>>>>>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>>> 3,
>>>>>>>>>>>>>>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Bas
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
>>>>>>> neil4real@gmail.com
>>>>>>>>>> <ma...@gmail.com>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG,
>>>>>>> especially
>>>>>>>> when
>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term
>> wholeheartedly.
>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or
>>>> confusing
>>>>>>>> about it
>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>> all,
>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly
>>>>>> confusing
>>>>>>>> to
>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>>>>> who
>>>>>>>>>>>>>>>>>>>>>>>>>> have been
>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
>>>> Chung
>>>>>>>>>>>>>>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
>>>>>>>> tp@astronomer.io.inva
>>>>>>>>>>> lid
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with
>> the
>>>> term
>>>>>>> DAG.
>>>>>>>> The
>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>>>>>>>>>>>> (aside
>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling peopleâincluding
>>>>>> myselfâto
>>>>>>>> stop
>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> term)
>>>>>>>>>>>>>>>>>>>>>>>>> is to
>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I canât recall who, but someone
>> mentioned
>>>>>>>> âworkflowâ is
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>>>>>>>>>>>> accurate
>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
>>>>>> proposed,
>>>>>>>> but
>>>>>>>>>>> itâs a
>>>>>>>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few
>> places
>>>> and
>>>>>> kept
>>>>>>>>>> finding
>>>>>>>>>>>>>>>>>>>> myself
>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting âwhat definition?â and
>> wanting to
>>>>>>> clarify
>>>>>>>> âDAG
>>>>>>>>>>>>>>>>>>>>>> definitionâ
>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose).
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> TP
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
>> Scheffler
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
>>>>>>>>>> j_scheffler@gmx.de.INVA>LID>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the
>> exactly
>>>> same
>>>>>>>>>>> observation,
>>>>>>>>>>>>>>>>>>>> had a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> short
>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is
>> always
>>>> an
>>>>>>>>>> introduction
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>> someone
>>>>>>>>>>>>>>>>>>>>>>>>> joins
>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming
>> makes
>>>>>> sense.
>>>>>>>>>>>>>>>>>> Especially
>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a
>>>>>> reasonable
>>>>>>>> step.
>>>>>>>>>>>>>>>>>>>> Concepts
>>>>>>>>>>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I donât need to join hiding
>>>> below
>>>>>> the
>>>>>>>> desk
>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we
>> keep
>>>>>>> details
>>>>>>>> of
>>>>>>>>>>>>>>>>>> python
>>>>>>>>>>>>>>>>>>>>> names
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a
>> DAGâŠ
>>>>> but
>>>>>>> user
>>>>>>>>>>> facing
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>> workflow.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
>> Hatter <
>>>>>>>>>>>>>>>>>>>>>> ryan.hatter@astronomer.io <mailto:
>>>>>>>> ryan.hatter@astronomer.io
>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> .invalid>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your
>> swords... at
>>>>>> least
>>>>>>>> for
>>>>>>>>>>> now.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little
>> meaning to
>>>>>>> Airflow
>>>>>>>>>>> users.
>>>>>>>>>>>>>>>>>>>>> Indeed,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some
>>>>> mathematicians
>>>>>>> and
>>>>>>>>>>>>>>>>>> software
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers
>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually
>>>>> matter.
>>>>>>> For
>>>>>>>>>>> someone
>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow
>>>> orchestration,
>>>>>> one
>>>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a
>> DAG?"
>>>>> The
>>>>>>>> answer
>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> almost
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic
>> graph.
>>>> You
>>>>>>> don't
>>>>>>>>>> need
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> worry
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your
>>>> workflow."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
>>>> least
>>>>> a
>>>>>>>> couple
>>>>>>>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>>>>>>> reasons:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As
>> mentioned
>>>>>> above,
>>>>>>>> "DAG"
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and
>> confusing.
>>>>> We
>>>>>>> want
>>>>>>>>>>> Airflow
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and
>>>>>>>>>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right
>> off the
>>>>> bat
>>>>>>>> creates
>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
>>>>>> Concepts*:
>>>>>>>> The
>>>>>>>>>> DAG
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow
>> workflow. The
>>>>>>>> workflow
>>>>>>>>>>>>>>>>>> includes
>>>>>>>>>>>>>>>>>>>>> its
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
>>>> other
>>>>>>>>>> parameters,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesnât account for.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the
>> Airflow
>>>>>>> homepage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
>>>>>>>>>> https://airflow.apache.org/>>.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
>> created by
>>>>> the
>>>>>>>>>> community
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor
>> workflows.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is
>>>> Airflow?"
>>>>>> docs
>>>>>>>> page
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
>>>>>>>>>>
>>>> https://airflow.apache.org/docs/apache-airflow/stable/index.html
>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what
>> Airflow is
>>>>>>> without
>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>>> "DAG."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>>>>>>>>> only in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the
>> term
>>>> is
>>>>>>>>>> introduced
>>>>>>>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere
>>>>>>>>>>>>>>>>>>>>>>>>>>> as a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to
>> explain
>>>> it:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
>>>> collection
>>>>>> of
>>>>>>>> tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
>> in
>>>>> these
>>>>>>>>>>>>>>>>>> introductions
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
>>>>> orchestrate
>>>>>>>> DAGs;
>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> orchestrates
>>>>>>>>>>>>>>>>>>>>>>>> workflows*.
>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons
>>>>> irrelevant
>>>>>>> to
>>>>>>>>>> almost
>>>>>>>>>>>>>>>>>>>> every
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an
>> alias
>>>> for
>>>>>> the
>>>>>>>> term
>>>>>>>>>>> "DAG"
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace
>> "DAG"
>>>>> with
>>>>>>>>>>> "workflow".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag(
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow(
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running
>>>> away.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>
>> ------------------------------------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
>>>>>>>>>> dev-unsubscribe@airflow.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>>>> dev-help@airflow.apache.org <mailto:
>>>>>>>>>> dev-help@airflow.apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Airflow should deprecate the term "DAG" for end users
Posted by Daniel Imberman <da...@gmail.com>.
I think my biggest concern is a marketing one and not a technical one.
As has been mentioned on the thread the terms airflow and dag are kind of
synonymous and I certainly donât want to give the impression that we are
breaking more than we are breaking.
I wouldnât die on this hill, but Iâm slightly concerned.
On Mon, Feb 17, 2025 at 6:00âŻPM Wei Lee <we...@apache.org> wrote:
> Iâm not sure about adding ruff rules here đ€ I think ruff rules are best
> suited for user-facing things but not the airflow code base itself. If what
> we mean is adding a rule to avoid users using "DAG" *after* we rename it,
> it's definitely a +1000.
>
> I just created GitHub issues for this removing "DAG"
>
> https://github.com/apache/airflow/issues/46842
> https://github.com/apache/airflow/issues/46843
>
> One thing I'm not sure about is whether we want to get rid of `from
> airflow import DAG` as well. đ€
>
> Best,
> Wei
>
> On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> > Hard to say until it's looked at :)
> >
> > On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <ar...@gmail.com>
> > wrote:
> >
> > > I can take it up, it's mostly just a doc update right? Or are we doing
> code
> > > files replacement too?
> > > --
> > > Regards,
> > > Aritra Basu
> > >
> > > On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com> wrote:
> > >
> > > > Sounds like another +1000 files big PR is coming :) ? Any volunteers
> to
> > > > make it ? It's fun.
> > > >
> > > > On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com>
> wrote:
> > > >
> > > > > +1 for ruff rules :)
> > > > >
> > > > > Also would be nice to introduce 'Dag' to replace 'DAG' in the
> Airflow
> > > > > docs, in line with the new UI changes and to make the renaming
> > > > > consistent across user-facing pages.
> > > > >
> > > > > Regards,
> > > > > Omkar
> > > > >
> > > > > On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > >
> > > > > > +1 :) . Maybe we could add a ruff rule for that :)
> > > > > >
> > > > > > On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org>
> wrote:
> > > > > >
> > > > > > > It seems that our current conclusion is to use "dag" or "Dag"
> > > instead
> > > > > of
> > > > > > > "DAG" whenever possible. Should we replace all "DAG" in the
> > > codebase
> > > > > with
> > > > > > > "dag" or "Dag"? If it's too late for that (which it might be
> đ€),
> > > > > should
> > > > > > we
> > > > > > > at least avoid introducing new "DAG" in the following PRs?
> > > > > > >
> > > > > > > Best,
> > > > > > > Wei
> > > > > > >
> > > > > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > > > > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > > > > > > https://github.com/apache/airflow/pull/43325
> > > > > > > > Let me know if that's the direction we want to go.
> > > > > > > >
> > > > > > > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > > > > > > <rb...@amazon.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > I think Brent & Daniel summarized it best, "dag" is
> synonymous
> > > > with
> > > > > > > > > workflows in Airflow through the way we talk and explain
> what
> > > > > Airflow
> > > > > > > is
> > > > > > > > > all about. Although folks would ask, I donât ever use the
> > > > > > mathematical
> > > > > > > > > definition of DAG.
> > > > > > > > > It will be challenging and possibly confusing for many
> users
> > > > making
> > > > > > > such a
> > > > > > > > > change - I would rather direct that energy to appending the
> > > > Oxford
> > > > > > > > > definition of "dag" to include a reference to workflows in
> > > > Airflow.
> > > > > > __
> > > > > > > > >
> > > > > > > > > -- Rajesh
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> > > > <mailto:
> > > > > > > > > jarek@potiuk.com>> wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > CAUTION: This email originated from outside of the
> > > organization.
> > > > Do
> > > > > > not
> > > > > > > > > click links or open attachments unless you can confirm the
> > > sender
> > > > > and
> > > > > > > know
> > > > > > > > > the content is safe.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > > expéditeur
> > > > > > > externe.
> > > > > > > > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
> si
> > > vous
> > > > > ne
> > > > > > > pouvez
> > > > > > > > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
> pas
> > > > > > certain
> > > > > > > que
> > > > > > > > > le contenu ne présente aucun risque.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > From Guido's post:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > "Around this time the renaming seems to have been renamed".
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Naming is hard.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > > droiddev510e@gmail.com
> > > > > > > <mailto:
> > > > > > > > > droiddev510e@gmail.com>> wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Yes, completely agree with above comments, dag is an
> Airflow
> > > > term
> > > > > > now
> > > > > > > > > > rather
> > > > > > > > > > than just a "directed acyclic graph". Believe it or not,
> I've
> > > > > > worked
> > > > > > > with
> > > > > > > > > > work
> > > > > > > > > > folks who've been using dags in Airflow for years now
> and are
> > > > pro
> > > > > > > devs
> > > > > > > > > but
> > > > > > > > > > have trouble remembering the full form of a DAG! For
> them,
> > > dag
> > > > =
> > > > > > > Airflow.
> > > > > > > > > >
> > > > > > > > > > workflow, pipeline, flow are surely better terms but
> will be
> > > a
> > > > > > major
> > > > > > > > > > behavior
> > > > > > > > > > change for the developer community, so we'll need a solid
> > > plan
> > > > on
> > > > > > > how to
> > > > > > > > > > introduce it, when we do.
> > > > > > > > > >
> > > > > > > > > > While we discuss this, I'd like to share about the Great
> > > > Renaming
> > > > > > in
> > > > > > > core
> > > > > > > > > > Python started by Guido back in 2009. We could probably
> get
> > > > some
> > > > > > > > > learnings
> > > > > > > > > > from there? Who knows!
> > > > > > > > > >
> > > > > > > > > > Guido's blog (2009):
> > > > > > > > > >
> > > > > > >
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > > > <
> > > > > > >
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > >
> > > > > > > > > > Follow-up discussion (2024):
> > > > > > > > > >
> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > > <
> > > > > > > > >
> > > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Omkar
> > > > > > > > > >
> > > > > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <
> weilee.rx@gmail.com
> > > > > > > <mailto:
> > > > > > > > > weilee.rx@gmail.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > > I think we should probably just accept it as an airflow
> > > term.
> > > > > At
> > > > > > > least,
> > > > > > > > > > > thatâs how I understood it when I first used Airflow. I
> > > feel
> > > > > > > renaming
> > > > > > > > > it
> > > > > > > > > > at
> > > > > > > > > > > this stage would require considerable effort from
> > > maintainers
> > > > > and
> > > > > > > > > > existing
> > > > > > > > > > > users without providing equivalent benefits.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Wei
> > > > > > > > > > >
> > > > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > > kaxilnaik@gmail.com
> > > > > > > > > <ma...@gmail.com>> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle
> > > this
> > > > > > > discussion
> > > > > > > > > > for
> > > > > > > > > > > > Airflow 4 :) -- but right now it will cause too much
> > > > > disruption
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > > > > > > > <constance@astronomer.io.inva <mailto:
> > > > > > > constance@astronomer.io.inva>lid>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> In my experience, when you ask those with Airflow
> > > > experience
> > > > > > > what a
> > > > > > > > > > dag
> > > > > > > > > > > is,
> > > > > > > > > > > >> they'll start talking about workflow attributes -
> stuff
> > > > like
> > > > > > > dags
> > > > > > > > > > being
> > > > > > > > > > > a
> > > > > > > > > > > >> series of steps or tasks with owners. The structure
> > > > doesn't
> > > > > > > come up.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Echo-ing others, at this point, my vote is to
> embrace
> > > the
> > > > > name
> > > > > > > and
> > > > > > > > > > > >> de-emphasize the mathematical structure aspect.
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > > > > > > <vikram@astronomer.io.inva <mailto:
> > > vikram@astronomer.io.inva
> > > > > > >lid>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> It's an interesting discussion and I remember
> > > struggling
> > > > > with
> > > > > > > this
> > > > > > > > > > > when I
> > > > > > > > > > > >>> started working with Airflow.
> > > > > > > > > > > >>> But, I also agree with the viewpoint of it being an
> > > > > > established
> > > > > > > > > > concept
> > > > > > > > > > > >> now
> > > > > > > > > > > >>> regardless of the origin.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> I am personally leaning towards the perspective
> best
> > > > > > expressed
> > > > > > > by
> > > > > > > > > > > Daniel
> > > > > > > > > > > >>> Standish and Brent of using Dag as a word, rather
> than
> > > > the
> > > > > > > computer
> > > > > > > > > > > >> science
> > > > > > > > > > > >>> concept.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Best regards,
> > > > > > > > > > > >>> Vikram
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > > > > > > > >> <onikolas@amazon.com.inva <mailto:
> > > > onikolas@amazon.com.inva
> > > > > > >lid
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> I agree with the general sentiment of: You're
> right
> > > > Ryan,
> > > > > > DAG
> > > > > > > > > isn't
> > > > > > > > > > > >> great
> > > > > > > > > > > >>>> and I'd rather workflow, but changing it will
> cause
> > > much
> > > > > > more
> > > > > > > > > > wreckage
> > > > > > > > > > > >>> than
> > > > > > > > > > > >>>> it solves.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Also agree with the idea to just move away from
> > > defining
> > > > > > DAG.
> > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > > >>>> we've been naturally doing that as a community
> for a
> > > > while
> > > > > > now
> > > > > > > > > > anyway,
> > > > > > > > > > > >> so
> > > > > > > > > > > >>>> that feels like a natural step.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Cheers,
> > > > > > > > > > > >>>> Niko
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> ________________________________
> > > > > > > > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > > > > > > ash@apache.org>>
> > > > > > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > > > > > > >>>> To: dev@airflow.apache.org <mailto:
> > > > dev@airflow.apache.org
> > > > > >
> > > > > > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the
> term
> > > > "DAG"
> > > > > > > for end
> > > > > > > > > > > >> users
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> CAUTION: This email originated from outside of the
> > > > > > > organization.
> > > > > > > > > Do
> > > > > > > > > > > not
> > > > > > > > > > > >>>> click links or open attachments unless you can
> confirm
> > > > the
> > > > > > > sender
> > > > > > > > > > and
> > > > > > > > > > > >>> know
> > > > > > > > > > > >>>> the content is safe.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> AVERTISSEMENT: Ce courrier électronique provient
> dâun
> > > > > > > expéditeur
> > > > > > > > > > > >> externe.
> > > > > > > > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > > > jointe
> > > > > si
> > > > > > > vous
> > > > > > > > > ne
> > > > > > > > > > > >>> pouvez
> > > > > > > > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si
> vous
> > > > nâĂȘtes
> > > > > > pas
> > > > > > > > > > certain
> > > > > > > > > > > >>> que
> > > > > > > > > > > >>>> le contenu ne présente aucun risque.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Best argument in favour of keeping âdagsâ as a
> term â
> > > > > > getting
> > > > > > > to
> > > > > > > > > > > re-use
> > > > > > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > > > > > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> In all seriousness: I donât mind either way, both
> > > sides
> > > > > have
> > > > > > > good
> > > > > > > > > > > >> reasons
> > > > > > > > > > > >>>> presented.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> -a
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > > > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > > > > > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Yeah just say, when asked where the name comes
> from,
> > > > > "well,
> > > > > > > no
> > > > > > > > > one
> > > > > > > > > > > >>>> actually
> > > > > > > > > > > >>>>> knows but..." and then make something up.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > > > > > > jarek@potiuk.com
> > > > > > > > > <ma...@potiuk.com>>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is
> the
> > > > > > > > > tongue-twister,
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > > > > > > jarek@potiuk.com
> > > > > > > > > <ma...@potiuk.com>>
> > > > > > > > > > > >>> wrote:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I
> would
> > > very
> > > > > > much
> > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > >> be
> > > > > > > > > > > >>>>>> able
> > > > > > > > > > > >>>>>>> to say just "dag" without explaining it
> further.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> For me every time I explain "DAG" at a talk
> it's a
> > > > > > > > > > tongue-twister,
> > > > > > > > > > > >>> and
> > > > > > > > > > > >>>> I
> > > > > > > > > > > >>>>>>> almost stutter on trying to recall how to
> pronounce
> > > > it
> > > > > > > > > properly.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> J.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > > > > > > > >>>>>> <brent@astronomer.io.inva <mailto:
> > > > > > brent@astronomer.io.inva
> > > > > > > >lid>
> > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> I remember we explored renaming "DAG" when
> > > starting
> > > > on
> > > > > > > AIP-38
> > > > > > > > > to
> > > > > > > > > > > >>>>>> modernize
> > > > > > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > > > > > > descriptive of
> > > > > > > > > > what
> > > > > > > > > > > >>> one
> > > > > > > > > > > >>>>>> is
> > > > > > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph
> is an
> > > > > > > > > implementation
> > > > > > > > > > > >>>> detail.
> > > > > > > > > > > >>>>>>>> But I agree with Daniel Standish, at this
> point
> > > > "DAG"
> > > > > > has
> > > > > > > > > become
> > > > > > > > > > > >>> "dag"
> > > > > > > > > > > >>>>>> , a
> > > > > > > > > > > >>>>>>>> word in its own right.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Examples for "dag" are abound in community
> > > > discussion,
> > > > > > > Airflow
> > > > > > > > > > > >>> Summit
> > > > > > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's
> > > > embrace
> > > > > > > "dag".
> > > > > > > > > A
> > > > > > > > > > > >> user
> > > > > > > > > > > >>>>>> just
> > > > > > > > > > > >>>>>>>> needs to learn one new word vs the technical
> > > concept
> > > > > > > behind
> > > > > > > > > that
> > > > > > > > > > > >>>> word. I
> > > > > > > > > > > >>>>>>>> think that is much less effort than
> refactoring so
> > > > > much
> > > > > > > code,
> > > > > > > > > > > >>>>>>>> documentation, blog posts, stack overflow
> > > questions,
> > > > > > etc.
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel
> Standish
> > > > > > > > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > > > > > > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot
> of
> > > > pain
> > > > > > for
> > > > > > > > > > > >>>> questionable
> > > > > > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the
> idea. I
> > > > > agree
> > > > > > > the
> > > > > > > > > > > >>>>>>>> association
> > > > > > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> And along those lines, I offer here some less
> > > > > invasive
> > > > > > > > > > > >> mitigations.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> One thing we can do no matter what is to
> > > > de-emphasize
> > > > > > the
> > > > > > > > > math
> > > > > > > > > > > >> nerd
> > > > > > > > > > > >>>>>>>> origins
> > > > > > > > > > > >>>>>>>>> of the name. That is to say, in docs /
> website /
> > > > etc,
> > > > > > > *never
> > > > > > > > > > > >>> define*
> > > > > > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic
> > > > graph.
> > > > > > > Always
> > > > > > > > > > > >> define
> > > > > > > > > > > >>>> it
> > > > > > > > > > > >>>>>>>> as a
> > > > > > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is
> like a
> > > > > > > historical
> > > > > > > > > > > >>>> footnote,
> > > > > > > > > > > >>>>>>>> and
> > > > > > > > > > > >>>>>>>>> we could make one mention of it somewhere
> hidden.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> We could also start using lowercase in the
> docs
> > > in
> > > > > > > general
> > > > > > > > > e.g.
> > > > > > > > > > > >>>>>> writing
> > > > > > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" /
> "DAGs"
> > > > etc.
> > > > > > The
> > > > > > > > > > upper
> > > > > > > > > > > >>> case
> > > > > > > > > > > >>>>>>>> part
> > > > > > > > > > > >>>>>>>>> of it makes it look like an acronym; but
> "dag" in
> > > > > > > airlfow is
> > > > > > > > > > just
> > > > > > > > > > > >>> an
> > > > > > > > > > > >>>>>>>>> airflow concept and the association with
> "DAGs"
> > > is
> > > > > not
> > > > > > > really
> > > > > > > > > > > >>>>>> unhelpful.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow
> is
> > > its
> > > > > own
> > > > > > > > > thing,
> > > > > > > > > > is
> > > > > > > > > > > >>>>>>>>> *not* strictly
> > > > > > > > > > > >>>>>>>>> speaking a directed acyclic graph (which
> nobody
> > > > knows
> > > > > > > about
> > > > > > > > > > > >>> anyway),
> > > > > > > > > > > >>>>>> and
> > > > > > > > > > > >>>>>>>>> tell them what it is in simple terms that
> normal
> > > > > people
> > > > > > > > > > > >> understand.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk
> <
> > > > > > > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > > > > > > >
> > > > > > > > > > > >>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it
> will
> > > be
> > > > > > > extremely
> > > > > > > > > > > >>>>>>>> difficult to
> > > > > > > > > > > >>>>>>>>>> get rid of it completely. Also I think it
> will
> > > > make
> > > > > a
> > > > > > > lot of
> > > > > > > > > > > >>>>>> "google"
> > > > > > > > > > > >>>>>>>>>> searches and "stack overflow" searches not
> > > finding
> > > > > the
> > > > > > > right
> > > > > > > > > > > >>>>>> answers.
> > > > > > > > > > > >>>>>>>>> This
> > > > > > > > > > > >>>>>>>>>> is one of the strengths of Airflow -
> besides the
> > > > > > > community
> > > > > > > > > and
> > > > > > > > > > > >>> ideas
> > > > > > > > > > > >>>>>>>> that
> > > > > > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of
> > > examples,
> > > > > > > problems
> > > > > > > > > and
> > > > > > > > > > > >>>>>>>> solutions
> > > > > > > > > > > >>>>>>>>>> you can so easily find (and we have to
> remember
> > > > that
> > > > > > > all the
> > > > > > > > > > AI
> > > > > > > > > > > >>>>>>>> trained
> > > > > > > > > > > >>>>>>>>> on
> > > > > > > > > > > >>>>>>>>>> past data will be also rather poorly
> matching
> > > > > queries
> > > > > > of
> > > > > > > > > > people.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily
> > > > switch.
> > > > > > > And if
> > > > > > > > > we
> > > > > > > > > > > >> do
> > > > > > > > > > > >>> -
> > > > > > > > > > > >>>>>> I
> > > > > > > > > > > >>>>>>>>>> would be for using workflow or pipeline
> instead
> > > of
> > > > > > > `dag` if
> > > > > > > > > > not
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>>>>>>> above
> > > > > > > > > > > >>>>>>>>>> reason, but I think I am here with Igor
> that it
> > > > > might
> > > > > > > cause
> > > > > > > > > > more
> > > > > > > > > > > >>>>>>>> problems
> > > > > > > > > > > >>>>>>>>>> than it solves.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> But I am not 100% against - if others will
> think
> > > > > it's
> > > > > > a
> > > > > > > good
> > > > > > > > > > > >>> idea, I
> > > > > > > > > > > >>>>>>>> am
> > > > > > > > > > > >>>>>>>>> ok
> > > > > > > > > > > >>>>>>>>>> with it.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> J,
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek
> Bhakat
> > > > > > > > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva
> <mailto:
> > > > > > > > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less
> sense
> > > to
> > > > > > > someone
> > > > > > > > > new
> > > > > > > > > > > >> to
> > > > > > > > > > > >>>>>>>>> workflow
> > > > > > > > > > > >>>>>>>>>>> orchestration. But it does also show the
> nature
> > > > of
> > > > > > > being
> > > > > > > > > > > >> acyclic.
> > > > > > > > > > > >>>>>>>> Sure,
> > > > > > > > > > > >>>>>>>>>> as
> > > > > > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to
> workaround it.
> > > > > > Still,
> > > > > > > in
> > > > > > > > > my
> > > > > > > > > > > >>>>>>>> opinion,
> > > > > > > > > > > >>>>>>>>>> there
> > > > > > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in
> > > > > workflow
> > > > > > > > > > > >>>>>> orchestration.
> > > > > > > > > > > >>>>>>>>> Most
> > > > > > > > > > > >>>>>>>>>>> (*if
> > > > > > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be
> > > covered
> > > > > > > using an
> > > > > > > > > > > >>> acyclic
> > > > > > > > > > > >>>>>>>>> manner
> > > > > > > > > > > >>>>>>>>>>> with multiple runs. Hence, the
> idempotency. So
> > > I
> > > > > > would
> > > > > > > want
> > > > > > > > > > the
> > > > > > > > > > > >>>>>>>>> "acyclic"
> > > > > > > > > > > >>>>>>>>>>> word to stick.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> Regards,
> > > > > > > > > > > >>>>>>>>>>> Avi
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > > > > > > bernd.stroehle@kosakya.de <mailto:
> bernd.stroehle@kosakya.de>>
> > > > > > > > > > > >>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an
> > > Airflow
> > > > > Fan;
> > > > > > > so
> > > > > > > > > many
> > > > > > > > > > > >> new
> > > > > > > > > > > >>>>>>>>> ideas.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be
> > > > replaced
> > > > > by
> > > > > > > the
> > > > > > > > > > more
> > > > > > > > > > > >>>>>>>> general
> > > > > > > > > > > >>>>>>>>>>> Term
> > > > > > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> > > > (Petri)
> > > > > > Net
> > > > > > > > > (AFN)
> > > > > > > > > > > >>>>>> (maybe
> > > > > > > > > > > >>>>>>>>>>> without
> > > > > > > > > > > >>>>>>>>>>>> a direction);
> > > > > > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a
> > > Graph
> > > > > > > Database.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow
> Graph
> > > > (AFG)
> > > > > > > might be
> > > > > > > > > > > >>>>>>>> assigned
> > > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>> an
> > > > > > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member
> of a
> > > > > > library.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> A running Graph might have a different
> > > structure
> > > > > > than
> > > > > > > a
> > > > > > > > > > > >>>>>>>> configuration
> > > > > > > > > > > >>>>>>>>>>>> Graph.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Best Regards
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > > > > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > > > > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > > > > > > bernd.stroehle@gmail.com>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > > > > > > > >>>>>>>>>>>> From: Igor Kholopov
> <ikholopov@google.com.INVA
> > > > > > > <mailto:
> > > > > > > > > ikholopov@google.com.INVA>LID>
> > > > > > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > > > dev@airflow.apache.org
> > > > > > > > > >
> > > > > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the
> term
> > > > > "DAG"
> > > > > > > for
> > > > > > > > > end
> > > > > > > > > > > >>>>>> users
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly
> > > > suboptimal,
> > > > > it
> > > > > > > is
> > > > > > > > > part
> > > > > > > > > > > >> of
> > > > > > > > > > > >>>>>>>>> Airflow
> > > > > > > > > > > >>>>>>>>>>>> DAG definition interface at so many
> levels,
> > > that
> > > > > any
> > > > > > > > > attempt
> > > > > > > > > > > >> to
> > > > > > > > > > > >>>>>>>>> change
> > > > > > > > > > > >>>>>>>>>> it
> > > > > > > > > > > >>>>>>>>>>>> will only introduce more chaos, not
> reduce it.
> > > > The
> > > > > > > only
> > > > > > > > > > thing
> > > > > > > > > > > >>>>>>>> that is
> > > > > > > > > > > >>>>>>>>>>> worse
> > > > > > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is
> when
> > > > > there
> > > > > > > are
> > > > > > > > > two
> > > > > > > > > > > >> ways
> > > > > > > > > > > >>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>> define
> > > > > > > > > > > >>>>>>>>>>>> the same thing. Countless articles and
> > > tutorials
> > > > > > will
> > > > > > > > > > suddenly
> > > > > > > > > > > >>>>>>>> become
> > > > > > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows
> as
> > > > > "DAG"s.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> We are already at risk of scaring the
> users
> > > away
> > > > > > with
> > > > > > > a
> > > > > > > > > > number
> > > > > > > > > > > >>>>>> of
> > > > > > > > > > > >>>>>>>>>>> breaking
> > > > > > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more
> > > > breaking
> > > > > > > changes
> > > > > > > > > > for
> > > > > > > > > > > >>>>>> the
> > > > > > > > > > > >>>>>>>>> most
> > > > > > > > > > > >>>>>>>>>>>> basic things is not something that people
> are
> > > > > > looking
> > > > > > > for.
> > > > > > > > > > > >>>>>>>> Attempting
> > > > > > > > > > > >>>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>> change the fundamental terms will be
> > > interpreted
> > > > > as
> > > > > > an
> > > > > > > > > even
> > > > > > > > > > > >>>>>>>> stronger
> > > > > > > > > > > >>>>>>>>>>> signal
> > > > > > > > > > > >>>>>>>>>>>> of project immaturity.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing
> the
> > > > term
> > > > > > in
> > > > > > > the
> > > > > > > > > > long
> > > > > > > > > > > >>>>>>>> run. I
> > > > > > > > > > > >>>>>>>>>>> even
> > > > > > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating
> it in
> > > > the
> > > > > > DAG
> > > > > > > > > > > >> definition
> > > > > > > > > > > >>>>>>>>>>> interface.
> > > > > > > > > > > >>>>>>>>>>>> We better put our time and efforts in
> other
> > > > places
> > > > > > in
> > > > > > > > > > Airflow,
> > > > > > > > > > > >>>>>> of
> > > > > > > > > > > >>>>>>>>> which
> > > > > > > > > > > >>>>>>>>>>>> there are plenty.
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> Kind regards,
> > > > > > > > > > > >>>>>>>>>>>> Igor
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas
> Harenslak
> > > > > > > > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > > > > > bas@astronomer.io.inva
> > > > > > > >lid
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ
> have
> > > > > already
> > > > > > > faded
> > > > > > > > > > over
> > > > > > > > > > > >>>>>>>> time,
> > > > > > > > > > > >>>>>>>>>>>>> for example there are now several ways to
> > > > create
> > > > > > > cyclic
> > > > > > > > > > > >>>>>> graphs,
> > > > > > > > > > > >>>>>>>>> e.g.
> > > > > > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine
> > > these
> > > > > > > > > properties
> > > > > > > > > > > >>>>>>>>> vanishing
> > > > > > > > > > > >>>>>>>>>>>>> even more in the future, so from that
> > > > > perspective I
> > > > > > > > > support
> > > > > > > > > > > >>>>>>>>> changing
> > > > > > > > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do
> > > > naming:
> > > > > > > > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > > > > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > > > > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > > > > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > > > > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting
> name.
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this
> change, I
> > > > > suggest
> > > > > > > > > > defining
> > > > > > > > > > > >> a
> > > > > > > > > > > >>>>>>>>> clear
> > > > > > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce
> the
> > > > > > > deprecation in
> > > > > > > > > > > >>>>>>>> Airflow
> > > > > > > > > > > >>>>>>>>> 3,
> > > > > > > > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>> Bas
> > > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > > > > > neil4real@gmail.com
> > > > > > > > > <ma...@gmail.com>>
> > > > > > > > > > wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > > > > > especially
> > > > > > > when
> > > > > > > > > > > >>>>>> most
> > > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > > >>>>>>>>>>>>>> platforms embrace the term
> wholeheartedly.
> > > > > > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or
> > > confusing
> > > > > > > about it
> > > > > > > > > at
> > > > > > > > > > > >>>>>>>> all,
> > > > > > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly
> > > > > confusing
> > > > > > > to
> > > > > > > > > most
> > > > > > > > > > > >>>>>>>> users
> > > > > > > > > > > >>>>>>>>>> who
> > > > > > > > > > > >>>>>>>>>>>>>> have been
> > > > > > > > > > > >>>>>>>>>>>>> using
> > > > > > > > > > > >>>>>>>>>>>>>> the term for years.
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> > > Chung
> > > > > > > > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > > > > > > tp@astronomer.io.inva
> > > > > > > > > >lid
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with
> the
> > > term
> > > > > > DAG.
> > > > > > > The
> > > > > > > > > > only
> > > > > > > > > > > >>>>>>>>>> problem
> > > > > > > > > > > >>>>>>>>>>>>> (aside
> > > > > > > > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding
> > > > > myselfâto
> > > > > > > stop
> > > > > > > > > > using
> > > > > > > > > > > >>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>>> term)
> > > > > > > > > > > >>>>>>>>>>>>> is to
> > > > > > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> I canât recall who, but someone
> mentioned
> > > > > > > âworkflowâ is
> > > > > > > > > > not
> > > > > > > > > > > >>>>>>>> very
> > > > > > > > > > > >>>>>>>>>>>>> accurate
> > > > > > > > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> > > > > proposed,
> > > > > > > but
> > > > > > > > > > itâs a
> > > > > > > > > > > >>>>>>>> bit
> > > > > > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few
> places
> > > and
> > > > > kept
> > > > > > > > > finding
> > > > > > > > > > > >>>>>>>> myself
> > > > > > > > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and
> wanting to
> > > > > > clarify
> > > > > > > âDAG
> > > > > > > > > > > >>>>>>>>>> definitionâ
> > > > > > > > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>> TP
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens
> Scheffler
> > > > > > > > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > > > > > > > j_scheffler@gmx.de.INVA>LID>
> > > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the
> exactly
> > > same
> > > > > > > > > > observation,
> > > > > > > > > > > >>>>>>>> had a
> > > > > > > > > > > >>>>>>>>>>>>>>>> short
> > > > > > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is
> always
> > > an
> > > > > > > > > introduction
> > > > > > > > > > > >>>>>> if
> > > > > > > > > > > >>>>>>>>>>>>>>> someone
> > > > > > > > > > > >>>>>>>>>>>>> joins
> > > > > > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming
> makes
> > > > > sense.
> > > > > > > > > > > >>>>>> Especially
> > > > > > > > > > > >>>>>>>>> when
> > > > > > > > > > > >>>>>>>>>>>>>>> we
> > > > > > > > > > > >>>>>>>>>>>>> also
> > > > > > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> > > > > reasonable
> > > > > > > step.
> > > > > > > > > > > >>>>>>>> Concepts
> > > > > > > > > > > >>>>>>>>>>>>>>> still
> > > > > > > > > > > >>>>>>>>>>>>> can
> > > > > > > > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> > > below
> > > > > the
> > > > > > > desk
> > > > > > > > > > with
> > > > > > > > > > > >>>>>>>> you
> > > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>>>>>> +1
> > > > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we
> keep
> > > > > > details
> > > > > > > of
> > > > > > > > > > > >>>>>> python
> > > > > > > > > > > >>>>>>>>> names
> > > > > > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>>> same because the execution is still a
> DAGâŠ
> > > > but
> > > > > > user
> > > > > > > > > > facing
> > > > > > > > > > > >>>>>> it
> > > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > > >>>>>>>>>>>>> workflow.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Jens
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan
> Hatter <
> > > > > > > > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > > > > > > ryan.hatter@astronomer.io
> > > > > > > > > >
> > > > > > > > > > > >>>>>>>>>>>>> .invalid>
> > > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your
> swords... at
> > > > > least
> > > > > > > for
> > > > > > > > > > now.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little
> meaning to
> > > > > > Airflow
> > > > > > > > > > users.
> > > > > > > > > > > >>>>>>>>> Indeed,
> > > > > > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > > > > > >>>>>>>>>>>>> has
> > > > > > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some
> > > > mathematicians
> > > > > > and
> > > > > > > > > > > >>>>>> software
> > > > > > > > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> > > > matter.
> > > > > > For
> > > > > > > > > > someone
> > > > > > > > > > > >>>>>>>> new
> > > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow
> > > orchestration,
> > > > > one
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > >>>>>>>> first
> > > > > > > > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > > > > > > > >>>>>>>>>>>>>>> will
> > > > > > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a
> DAG?"
> > > > The
> > > > > > > answer
> > > > > > > > > is
> > > > > > > > > > > >>>>>>>> almost
> > > > > > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic
> graph.
> > > You
> > > > > > don't
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > >>>>>>>>> worry
> > > > > > > > > > > >>>>>>>>>>>>>>>>> about what
> > > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your
> > > workflow."
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> > > least
> > > > a
> > > > > > > couple
> > > > > > > > > > > >>>>>>>> important
> > > > > > > > > > > >>>>>>>>>>>> reasons:
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As
> mentioned
> > > > > above,
> > > > > > > "DAG"
> > > > > > > > > > is
> > > > > > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and
> confusing.
> > > > We
> > > > > > want
> > > > > > > > > > Airflow
> > > > > > > > > > > >>>>>>>> to
> > > > > > > > > > > >>>>>>>>> be
> > > > > > > > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > > > > > > > >>>>>>>>>>>>>>> using
> > > > > > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right
> off the
> > > > bat
> > > > > > > creates
> > > > > > > > > > an
> > > > > > > > > > > >>>>>>>>> initial
> > > > > > > > > > > >>>>>>>>>>>>>>> barrier to
> > > > > > > > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> > > > > Concepts*:
> > > > > > > The
> > > > > > > > > DAG
> > > > > > > > > > > >>>>>> is
> > > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow
> workflow. The
> > > > > > > workflow
> > > > > > > > > > > >>>>>> includes
> > > > > > > > > > > >>>>>>>>> its
> > > > > > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> > > other
> > > > > > > > > parameters,
> > > > > > > > > > > >>>>>> and
> > > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the
> Airflow
> > > > > > homepage
> > > > > > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > > > > > > > https://airflow.apache.org/>>.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform
> created by
> > > > the
> > > > > > > > > community
> > > > > > > > > > to
> > > > > > > > > > > >>>>>>>>>>>>>>> programmatically
> > > > > > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor
> workflows.
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is
> > > Airflow?"
> > > > > docs
> > > > > > > page
> > > > > > > > > > > >>>>>>>>>>>>>>>>> <
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>
> > > > > > >
> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > > > > > > >
> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > > > >
> > > > > > > > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > > > > > > > >>>>>>>>>>>>> we
> > > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what
> Airflow is
> > > > > > without
> > > > > > > > > using
> > > > > > > > > > > >>>>>>>> "DAG."
> > > > > > > > > > > >>>>>>>>>>>>>>>>> It's
> > > > > > > > > > > >>>>>>>>>>>>>>> only in
> > > > > > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the
> term
> > > is
> > > > > > > > > introduced
> > > > > > > > > > > >>>>>> out
> > > > > > > > > > > >>>>>>>> of
> > > > > > > > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > > > > > > > >>>>>>>>>>>>>>> as a
> > > > > > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to
> explain
> > > it:
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> > > collection
> > > > > of
> > > > > > > tasks
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs
> in
> > > > these
> > > > > > > > > > > >>>>>> introductions
> > > > > > > > > > > >>>>>>>> to
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> > > > orchestrate
> > > > > > > DAGs;
> > > > > > > > > it
> > > > > > > > > > > >>>>>>>>>>> orchestrates
> > > > > > > > > > > >>>>>>>>>>>> workflows*.
> > > > > > > > > > > >>>>>>>>>>>>>>> The
> > > > > > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> > > > irrelevant
> > > > > > to
> > > > > > > > > almost
> > > > > > > > > > > >>>>>>>> every
> > > > > > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an
> alias
> > > for
> > > > > the
> > > > > > > term
> > > > > > > > > > "DAG"
> > > > > > > > > > > >>>>>>>> and
> > > > > > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace
> "DAG"
> > > > with
> > > > > > > > > > "workflow".
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running
> > > away.
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>
> > > > > > >
> ------------------------------------------------------------------
> > > > > > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > > > > > > > dev-help@airflow.apache.org>
> > > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Wei Lee <we...@apache.org>.
Iâm not sure about adding ruff rules here đ€ I think ruff rules are best suited for user-facing things but not the airflow code base itself. If what we mean is adding a rule to avoid users using "DAG" *after* we rename it, it's definitely a +1000.
I just created GitHub issues for this removing "DAG"
https://github.com/apache/airflow/issues/46842
https://github.com/apache/airflow/issues/46843
One thing I'm not sure about is whether we want to get rid of `from airflow import DAG` as well. đ€
Best,
Wei
On 2025/02/17 19:50:57 Jarek Potiuk wrote:
> Hard to say until it's looked at :)
>
> On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <ar...@gmail.com>
> wrote:
>
> > I can take it up, it's mostly just a doc update right? Or are we doing code
> > files replacement too?
> > --
> > Regards,
> > Aritra Basu
> >
> > On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com> wrote:
> >
> > > Sounds like another +1000 files big PR is coming :) ? Any volunteers to
> > > make it ? It's fun.
> > >
> > > On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com> wrote:
> > >
> > > > +1 for ruff rules :)
> > > >
> > > > Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow
> > > > docs, in line with the new UI changes and to make the renaming
> > > > consistent across user-facing pages.
> > > >
> > > > Regards,
> > > > Omkar
> > > >
> > > > On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > > +1 :) . Maybe we could add a ruff rule for that :)
> > > > >
> > > > > On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org> wrote:
> > > > >
> > > > > > It seems that our current conclusion is to use "dag" or "Dag"
> > instead
> > > > of
> > > > > > "DAG" whenever possible. Should we replace all "DAG" in the
> > codebase
> > > > with
> > > > > > "dag" or "Dag"? If it's too late for that (which it might be đ€),
> > > > should
> > > > > we
> > > > > > at least avoid introducing new "DAG" in the following PRs?
> > > > > >
> > > > > > Best,
> > > > > > Wei
> > > > > >
> > > > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > > > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > > > > > https://github.com/apache/airflow/pull/43325
> > > > > > > Let me know if that's the direction we want to go.
> > > > > > >
> > > > > > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > > > > > <rb...@amazon.com.invalid> wrote:
> > > > > > >
> > > > > > > > I think Brent & Daniel summarized it best, "dag" is synonymous
> > > with
> > > > > > > > workflows in Airflow through the way we talk and explain what
> > > > Airflow
> > > > > > is
> > > > > > > > all about. Although folks would ask, I donât ever use the
> > > > > mathematical
> > > > > > > > definition of DAG.
> > > > > > > > It will be challenging and possibly confusing for many users
> > > making
> > > > > > such a
> > > > > > > > change - I would rather direct that energy to appending the
> > > Oxford
> > > > > > > > definition of "dag" to include a reference to workflows in
> > > Airflow.
> > > > > __
> > > > > > > >
> > > > > > > > -- Rajesh
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> > > <mailto:
> > > > > > > > jarek@potiuk.com>> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > CAUTION: This email originated from outside of the
> > organization.
> > > Do
> > > > > not
> > > > > > > > click links or open attachments unless you can confirm the
> > sender
> > > > and
> > > > > > know
> > > > > > > > the content is safe.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > expéditeur
> > > > > > externe.
> > > > > > > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si
> > vous
> > > > ne
> > > > > > pouvez
> > > > > > > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> > > > > certain
> > > > > > que
> > > > > > > > le contenu ne présente aucun risque.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > From Guido's post:
> > > > > > > >
> > > > > > > >
> > > > > > > > "Around this time the renaming seems to have been renamed".
> > > > > > > >
> > > > > > > >
> > > > > > > > Naming is hard.
> > > > > > > >
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> > droiddev510e@gmail.com
> > > > > > <mailto:
> > > > > > > > droiddev510e@gmail.com>> wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > > Yes, completely agree with above comments, dag is an Airflow
> > > term
> > > > > now
> > > > > > > > > rather
> > > > > > > > > than just a "directed acyclic graph". Believe it or not, I've
> > > > > worked
> > > > > > with
> > > > > > > > > work
> > > > > > > > > folks who've been using dags in Airflow for years now and are
> > > pro
> > > > > > devs
> > > > > > > > but
> > > > > > > > > have trouble remembering the full form of a DAG! For them,
> > dag
> > > =
> > > > > > Airflow.
> > > > > > > > >
> > > > > > > > > workflow, pipeline, flow are surely better terms but will be
> > a
> > > > > major
> > > > > > > > > behavior
> > > > > > > > > change for the developer community, so we'll need a solid
> > plan
> > > on
> > > > > > how to
> > > > > > > > > introduce it, when we do.
> > > > > > > > >
> > > > > > > > > While we discuss this, I'd like to share about the Great
> > > Renaming
> > > > > in
> > > > > > core
> > > > > > > > > Python started by Guido back in 2009. We could probably get
> > > some
> > > > > > > > learnings
> > > > > > > > > from there? Who knows!
> > > > > > > > >
> > > > > > > > > Guido's blog (2009):
> > > > > > > > >
> > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > > <
> > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > >
> > > > > > > > > Follow-up discussion (2024):
> > > > > > > > >
> > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > > <
> > > > > > > >
> > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Omkar
> > > > > > > > >
> > > > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com
> > > > > > <mailto:
> > > > > > > > weilee.rx@gmail.com>> wrote:
> > > > > > > > >
> > > > > > > > > > I think we should probably just accept it as an airflow
> > term.
> > > > At
> > > > > > least,
> > > > > > > > > > thatâs how I understood it when I first used Airflow. I
> > feel
> > > > > > renaming
> > > > > > > > it
> > > > > > > > > at
> > > > > > > > > > this stage would require considerable effort from
> > maintainers
> > > > and
> > > > > > > > > existing
> > > > > > > > > > users without providing equivalent benefits.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Wei
> > > > > > > > > >
> > > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > > kaxilnaik@gmail.com
> > > > > > > > <ma...@gmail.com>> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle
> > this
> > > > > > discussion
> > > > > > > > > for
> > > > > > > > > > > Airflow 4 :) -- but right now it will cause too much
> > > > disruption
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > > > > > > <constance@astronomer.io.inva <mailto:
> > > > > > constance@astronomer.io.inva>lid>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> In my experience, when you ask those with Airflow
> > > experience
> > > > > > what a
> > > > > > > > > dag
> > > > > > > > > > is,
> > > > > > > > > > >> they'll start talking about workflow attributes - stuff
> > > like
> > > > > > dags
> > > > > > > > > being
> > > > > > > > > > a
> > > > > > > > > > >> series of steps or tasks with owners. The structure
> > > doesn't
> > > > > > come up.
> > > > > > > > > > >>
> > > > > > > > > > >> Echo-ing others, at this point, my vote is to embrace
> > the
> > > > name
> > > > > > and
> > > > > > > > > > >> de-emphasize the mathematical structure aspect.
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > > > > > <vikram@astronomer.io.inva <mailto:
> > vikram@astronomer.io.inva
> > > > > >lid>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> It's an interesting discussion and I remember
> > struggling
> > > > with
> > > > > > this
> > > > > > > > > > when I
> > > > > > > > > > >>> started working with Airflow.
> > > > > > > > > > >>> But, I also agree with the viewpoint of it being an
> > > > > established
> > > > > > > > > concept
> > > > > > > > > > >> now
> > > > > > > > > > >>> regardless of the origin.
> > > > > > > > > > >>>
> > > > > > > > > > >>> I am personally leaning towards the perspective best
> > > > > expressed
> > > > > > by
> > > > > > > > > > Daniel
> > > > > > > > > > >>> Standish and Brent of using Dag as a word, rather than
> > > the
> > > > > > computer
> > > > > > > > > > >> science
> > > > > > > > > > >>> concept.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Best regards,
> > > > > > > > > > >>> Vikram
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > > > > > > >> <onikolas@amazon.com.inva <mailto:
> > > onikolas@amazon.com.inva
> > > > > >lid
> > > > > > > > > > >>>>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> I agree with the general sentiment of: You're right
> > > Ryan,
> > > > > DAG
> > > > > > > > isn't
> > > > > > > > > > >> great
> > > > > > > > > > >>>> and I'd rather workflow, but changing it will cause
> > much
> > > > > more
> > > > > > > > > wreckage
> > > > > > > > > > >>> than
> > > > > > > > > > >>>> it solves.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Also agree with the idea to just move away from
> > defining
> > > > > DAG.
> > > > > > I
> > > > > > > > > think
> > > > > > > > > > >>>> we've been naturally doing that as a community for a
> > > while
> > > > > now
> > > > > > > > > anyway,
> > > > > > > > > > >> so
> > > > > > > > > > >>>> that feels like a natural step.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Cheers,
> > > > > > > > > > >>>> Niko
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> ________________________________
> > > > > > > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > > > > > ash@apache.org>>
> > > > > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > > > > > >>>> To: dev@airflow.apache.org <mailto:
> > > dev@airflow.apache.org
> > > > >
> > > > > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term
> > > "DAG"
> > > > > > for end
> > > > > > > > > > >> users
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> CAUTION: This email originated from outside of the
> > > > > > organization.
> > > > > > > > Do
> > > > > > > > > > not
> > > > > > > > > > >>>> click links or open attachments unless you can confirm
> > > the
> > > > > > sender
> > > > > > > > > and
> > > > > > > > > > >>> know
> > > > > > > > > > >>>> the content is safe.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > > > > > expéditeur
> > > > > > > > > > >> externe.
> > > > > > > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > > jointe
> > > > si
> > > > > > vous
> > > > > > > > ne
> > > > > > > > > > >>> pouvez
> > > > > > > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous
> > > nâĂȘtes
> > > > > pas
> > > > > > > > > certain
> > > > > > > > > > >>> que
> > > > > > > > > > >>>> le contenu ne présente aucun risque.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Best argument in favour of keeping âdagsâ as a term â
> > > > > getting
> > > > > > to
> > > > > > > > > > re-use
> > > > > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > > > > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> In all seriousness: I donât mind either way, both
> > sides
> > > > have
> > > > > > good
> > > > > > > > > > >> reasons
> > > > > > > > > > >>>> presented.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> -a
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > > > > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Yeah just say, when asked where the name comes from,
> > > > "well,
> > > > > > no
> > > > > > > > one
> > > > > > > > > > >>>> actually
> > > > > > > > > > >>>>> knows but..." and then make something up.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > > > > > jarek@potiuk.com
> > > > > > > > <ma...@potiuk.com>>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > > > > > > > tongue-twister,
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > > > > > jarek@potiuk.com
> > > > > > > > <ma...@potiuk.com>>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would
> > very
> > > > > much
> > > > > > want
> > > > > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>> able
> > > > > > > > > > >>>>>>> to say just "dag" without explaining it further.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > > > > > > > tongue-twister,
> > > > > > > > > > >>> and
> > > > > > > > > > >>>> I
> > > > > > > > > > >>>>>>> almost stutter on trying to recall how to pronounce
> > > it
> > > > > > > > properly.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> J.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > > > > > > >>>>>> <brent@astronomer.io.inva <mailto:
> > > > > brent@astronomer.io.inva
> > > > > > >lid>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> I remember we explored renaming "DAG" when
> > starting
> > > on
> > > > > > AIP-38
> > > > > > > > to
> > > > > > > > > > >>>>>> modernize
> > > > > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > > > > > descriptive of
> > > > > > > > > what
> > > > > > > > > > >>> one
> > > > > > > > > > >>>>>> is
> > > > > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > > > > > > > implementation
> > > > > > > > > > >>>> detail.
> > > > > > > > > > >>>>>>>> But I agree with Daniel Standish, at this point
> > > "DAG"
> > > > > has
> > > > > > > > become
> > > > > > > > > > >>> "dag"
> > > > > > > > > > >>>>>> , a
> > > > > > > > > > >>>>>>>> word in its own right.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Examples for "dag" are abound in community
> > > discussion,
> > > > > > Airflow
> > > > > > > > > > >>> Summit
> > > > > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's
> > > embrace
> > > > > > "dag".
> > > > > > > > A
> > > > > > > > > > >> user
> > > > > > > > > > >>>>>> just
> > > > > > > > > > >>>>>>>> needs to learn one new word vs the technical
> > concept
> > > > > > behind
> > > > > > > > that
> > > > > > > > > > >>>> word. I
> > > > > > > > > > >>>>>>>> think that is much less effort than refactoring so
> > > > much
> > > > > > code,
> > > > > > > > > > >>>>>>>> documentation, blog posts, stack overflow
> > questions,
> > > > > etc.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > > > > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > > > > > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of
> > > pain
> > > > > for
> > > > > > > > > > >>>> questionable
> > > > > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I
> > > > agree
> > > > > > the
> > > > > > > > > > >>>>>>>> association
> > > > > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> And along those lines, I offer here some less
> > > > invasive
> > > > > > > > > > >> mitigations.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> One thing we can do no matter what is to
> > > de-emphasize
> > > > > the
> > > > > > > > math
> > > > > > > > > > >> nerd
> > > > > > > > > > >>>>>>>> origins
> > > > > > > > > > >>>>>>>>> of the name. That is to say, in docs / website /
> > > etc,
> > > > > > *never
> > > > > > > > > > >>> define*
> > > > > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic
> > > graph.
> > > > > > Always
> > > > > > > > > > >> define
> > > > > > > > > > >>>> it
> > > > > > > > > > >>>>>>>> as a
> > > > > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a
> > > > > > historical
> > > > > > > > > > >>>> footnote,
> > > > > > > > > > >>>>>>>> and
> > > > > > > > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> We could also start using lowercase in the docs
> > in
> > > > > > general
> > > > > > > > e.g.
> > > > > > > > > > >>>>>> writing
> > > > > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs"
> > > etc.
> > > > > The
> > > > > > > > > upper
> > > > > > > > > > >>> case
> > > > > > > > > > >>>>>>>> part
> > > > > > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in
> > > > > > airlfow is
> > > > > > > > > just
> > > > > > > > > > >>> an
> > > > > > > > > > >>>>>>>>> airflow concept and the association with "DAGs"
> > is
> > > > not
> > > > > > really
> > > > > > > > > > >>>>>> unhelpful.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is
> > its
> > > > own
> > > > > > > > thing,
> > > > > > > > > is
> > > > > > > > > > >>>>>>>>> *not* strictly
> > > > > > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody
> > > knows
> > > > > > about
> > > > > > > > > > >>> anyway),
> > > > > > > > > > >>>>>> and
> > > > > > > > > > >>>>>>>>> tell them what it is in simple terms that normal
> > > > people
> > > > > > > > > > >> understand.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > > > > > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > > > > > >
> > > > > > > > > > >>>>>> wrote:
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will
> > be
> > > > > > extremely
> > > > > > > > > > >>>>>>>> difficult to
> > > > > > > > > > >>>>>>>>>> get rid of it completely. Also I think it will
> > > make
> > > > a
> > > > > > lot of
> > > > > > > > > > >>>>>> "google"
> > > > > > > > > > >>>>>>>>>> searches and "stack overflow" searches not
> > finding
> > > > the
> > > > > > right
> > > > > > > > > > >>>>>> answers.
> > > > > > > > > > >>>>>>>>> This
> > > > > > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the
> > > > > > community
> > > > > > > > and
> > > > > > > > > > >>> ideas
> > > > > > > > > > >>>>>>>> that
> > > > > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of
> > examples,
> > > > > > problems
> > > > > > > > and
> > > > > > > > > > >>>>>>>> solutions
> > > > > > > > > > >>>>>>>>>> you can so easily find (and we have to remember
> > > that
> > > > > > all the
> > > > > > > > > AI
> > > > > > > > > > >>>>>>>> trained
> > > > > > > > > > >>>>>>>>> on
> > > > > > > > > > >>>>>>>>>> past data will be also rather poorly matching
> > > > queries
> > > > > of
> > > > > > > > > people.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily
> > > switch.
> > > > > > And if
> > > > > > > > we
> > > > > > > > > > >> do
> > > > > > > > > > >>> -
> > > > > > > > > > >>>>>> I
> > > > > > > > > > >>>>>>>>>> would be for using workflow or pipeline instead
> > of
> > > > > > `dag` if
> > > > > > > > > not
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>>> above
> > > > > > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it
> > > > might
> > > > > > cause
> > > > > > > > > more
> > > > > > > > > > >>>>>>>> problems
> > > > > > > > > > >>>>>>>>>> than it solves.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> But I am not 100% against - if others will think
> > > > it's
> > > > > a
> > > > > > good
> > > > > > > > > > >>> idea, I
> > > > > > > > > > >>>>>>>> am
> > > > > > > > > > >>>>>>>>> ok
> > > > > > > > > > >>>>>>>>>> with it.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> J,
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > > > > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > > > > > > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense
> > to
> > > > > > someone
> > > > > > > > new
> > > > > > > > > > >> to
> > > > > > > > > > >>>>>>>>> workflow
> > > > > > > > > > >>>>>>>>>>> orchestration. But it does also show the nature
> > > of
> > > > > > being
> > > > > > > > > > >> acyclic.
> > > > > > > > > > >>>>>>>> Sure,
> > > > > > > > > > >>>>>>>>>> as
> > > > > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it.
> > > > > Still,
> > > > > > in
> > > > > > > > my
> > > > > > > > > > >>>>>>>> opinion,
> > > > > > > > > > >>>>>>>>>> there
> > > > > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in
> > > > workflow
> > > > > > > > > > >>>>>> orchestration.
> > > > > > > > > > >>>>>>>>> Most
> > > > > > > > > > >>>>>>>>>>> (*if
> > > > > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be
> > covered
> > > > > > using an
> > > > > > > > > > >>> acyclic
> > > > > > > > > > >>>>>>>>> manner
> > > > > > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So
> > I
> > > > > would
> > > > > > want
> > > > > > > > > the
> > > > > > > > > > >>>>>>>>> "acyclic"
> > > > > > > > > > >>>>>>>>>>> word to stick.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> Regards,
> > > > > > > > > > >>>>>>>>>>> Avi
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > > > > > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > > > > > > > >>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an
> > Airflow
> > > > Fan;
> > > > > > so
> > > > > > > > many
> > > > > > > > > > >> new
> > > > > > > > > > >>>>>>>>> ideas.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be
> > > replaced
> > > > by
> > > > > > the
> > > > > > > > > more
> > > > > > > > > > >>>>>>>> general
> > > > > > > > > > >>>>>>>>>>> Term
> > > > > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> > > (Petri)
> > > > > Net
> > > > > > > > (AFN)
> > > > > > > > > > >>>>>> (maybe
> > > > > > > > > > >>>>>>>>>>> without
> > > > > > > > > > >>>>>>>>>>>> a direction);
> > > > > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a
> > Graph
> > > > > > Database.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph
> > > (AFG)
> > > > > > might be
> > > > > > > > > > >>>>>>>> assigned
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>> an
> > > > > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a
> > > > > library.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> A running Graph might have a different
> > structure
> > > > > than
> > > > > > a
> > > > > > > > > > >>>>>>>> configuration
> > > > > > > > > > >>>>>>>>>>>> Graph.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Best Regards
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > > > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > > > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > > > > > bernd.stroehle@gmail.com>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA
> > > > > > <mailto:
> > > > > > > > ikholopov@google.com.INVA>LID>
> > > > > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > > dev@airflow.apache.org
> > > > > > > > >
> > > > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term
> > > > "DAG"
> > > > > > for
> > > > > > > > end
> > > > > > > > > > >>>>>> users
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly
> > > suboptimal,
> > > > it
> > > > > > is
> > > > > > > > part
> > > > > > > > > > >> of
> > > > > > > > > > >>>>>>>>> Airflow
> > > > > > > > > > >>>>>>>>>>>> DAG definition interface at so many levels,
> > that
> > > > any
> > > > > > > > attempt
> > > > > > > > > > >> to
> > > > > > > > > > >>>>>>>>> change
> > > > > > > > > > >>>>>>>>>> it
> > > > > > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it.
> > > The
> > > > > > only
> > > > > > > > > thing
> > > > > > > > > > >>>>>>>> that is
> > > > > > > > > > >>>>>>>>>>> worse
> > > > > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when
> > > > there
> > > > > > are
> > > > > > > > two
> > > > > > > > > > >> ways
> > > > > > > > > > >>>>>>>> to
> > > > > > > > > > >>>>>>>>>>> define
> > > > > > > > > > >>>>>>>>>>>> the same thing. Countless articles and
> > tutorials
> > > > > will
> > > > > > > > > suddenly
> > > > > > > > > > >>>>>>>> become
> > > > > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as
> > > > "DAG"s.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> We are already at risk of scaring the users
> > away
> > > > > with
> > > > > > a
> > > > > > > > > number
> > > > > > > > > > >>>>>> of
> > > > > > > > > > >>>>>>>>>>> breaking
> > > > > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more
> > > breaking
> > > > > > changes
> > > > > > > > > for
> > > > > > > > > > >>>>>> the
> > > > > > > > > > >>>>>>>>> most
> > > > > > > > > > >>>>>>>>>>>> basic things is not something that people are
> > > > > looking
> > > > > > for.
> > > > > > > > > > >>>>>>>> Attempting
> > > > > > > > > > >>>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>> change the fundamental terms will be
> > interpreted
> > > > as
> > > > > an
> > > > > > > > even
> > > > > > > > > > >>>>>>>> stronger
> > > > > > > > > > >>>>>>>>>>> signal
> > > > > > > > > > >>>>>>>>>>>> of project immaturity.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the
> > > term
> > > > > in
> > > > > > the
> > > > > > > > > long
> > > > > > > > > > >>>>>>>> run. I
> > > > > > > > > > >>>>>>>>>>> even
> > > > > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in
> > > the
> > > > > DAG
> > > > > > > > > > >> definition
> > > > > > > > > > >>>>>>>>>>> interface.
> > > > > > > > > > >>>>>>>>>>>> We better put our time and efforts in other
> > > places
> > > > > in
> > > > > > > > > Airflow,
> > > > > > > > > > >>>>>> of
> > > > > > > > > > >>>>>>>>> which
> > > > > > > > > > >>>>>>>>>>>> there are plenty.
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> Kind regards,
> > > > > > > > > > >>>>>>>>>>>> Igor
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > > > > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > > > > bas@astronomer.io.inva
> > > > > > >lid
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have
> > > > already
> > > > > > faded
> > > > > > > > > over
> > > > > > > > > > >>>>>>>> time,
> > > > > > > > > > >>>>>>>>>>>>> for example there are now several ways to
> > > create
> > > > > > cyclic
> > > > > > > > > > >>>>>> graphs,
> > > > > > > > > > >>>>>>>>> e.g.
> > > > > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine
> > these
> > > > > > > > properties
> > > > > > > > > > >>>>>>>>> vanishing
> > > > > > > > > > >>>>>>>>>>>>> even more in the future, so from that
> > > > perspective I
> > > > > > > > support
> > > > > > > > > > >>>>>>>>> changing
> > > > > > > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do
> > > naming:
> > > > > > > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > > > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > > > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > > > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > > > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I
> > > > suggest
> > > > > > > > > defining
> > > > > > > > > > >> a
> > > > > > > > > > >>>>>>>>> clear
> > > > > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the
> > > > > > deprecation in
> > > > > > > > > > >>>>>>>> Airflow
> > > > > > > > > > >>>>>>>>> 3,
> > > > > > > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>> Bas
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > > > > neil4real@gmail.com
> > > > > > > > <ma...@gmail.com>>
> > > > > > > > > wrote:
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > > > > especially
> > > > > > when
> > > > > > > > > > >>>>>> most
> > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or
> > confusing
> > > > > > about it
> > > > > > > > at
> > > > > > > > > > >>>>>>>> all,
> > > > > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly
> > > > confusing
> > > > > > to
> > > > > > > > most
> > > > > > > > > > >>>>>>>> users
> > > > > > > > > > >>>>>>>>>> who
> > > > > > > > > > >>>>>>>>>>>>>> have been
> > > > > > > > > > >>>>>>>>>>>>> using
> > > > > > > > > > >>>>>>>>>>>>>> the term for years.
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> > Chung
> > > > > > > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > > > > > tp@astronomer.io.inva
> > > > > > > > >lid
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the
> > term
> > > > > DAG.
> > > > > > The
> > > > > > > > > only
> > > > > > > > > > >>>>>>>>>> problem
> > > > > > > > > > >>>>>>>>>>>>> (aside
> > > > > > > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding
> > > > myselfâto
> > > > > > stop
> > > > > > > > > using
> > > > > > > > > > >>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>>> term)
> > > > > > > > > > >>>>>>>>>>>>> is to
> > > > > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned
> > > > > > âworkflowâ is
> > > > > > > > > not
> > > > > > > > > > >>>>>>>> very
> > > > > > > > > > >>>>>>>>>>>>> accurate
> > > > > > > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> > > > proposed,
> > > > > > but
> > > > > > > > > itâs a
> > > > > > > > > > >>>>>>>> bit
> > > > > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places
> > and
> > > > kept
> > > > > > > > finding
> > > > > > > > > > >>>>>>>> myself
> > > > > > > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to
> > > > > clarify
> > > > > > âDAG
> > > > > > > > > > >>>>>>>>>> definitionâ
> > > > > > > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>> TP
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > > > > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > > > > > > j_scheffler@gmx.de.INVA>LID>
> > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly
> > same
> > > > > > > > > observation,
> > > > > > > > > > >>>>>>>> had a
> > > > > > > > > > >>>>>>>>>>>>>>>> short
> > > > > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always
> > an
> > > > > > > > introduction
> > > > > > > > > > >>>>>> if
> > > > > > > > > > >>>>>>>>>>>>>>> someone
> > > > > > > > > > >>>>>>>>>>>>> joins
> > > > > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes
> > > > sense.
> > > > > > > > > > >>>>>> Especially
> > > > > > > > > > >>>>>>>>> when
> > > > > > > > > > >>>>>>>>>>>>>>> we
> > > > > > > > > > >>>>>>>>>>>>> also
> > > > > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> > > > reasonable
> > > > > > step.
> > > > > > > > > > >>>>>>>> Concepts
> > > > > > > > > > >>>>>>>>>>>>>>> still
> > > > > > > > > > >>>>>>>>>>>>> can
> > > > > > > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> > below
> > > > the
> > > > > > desk
> > > > > > > > > with
> > > > > > > > > > >>>>>>>> you
> > > > > > > > > > >>>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>>>>>> +1
> > > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep
> > > > > details
> > > > > > of
> > > > > > > > > > >>>>>> python
> > > > > > > > > > >>>>>>>>> names
> > > > > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAGâŠ
> > > but
> > > > > user
> > > > > > > > > facing
> > > > > > > > > > >>>>>> it
> > > > > > > > > > >>>>>>>>> is a
> > > > > > > > > > >>>>>>>>>>>>> workflow.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Jens
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > > > > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > > > > > ryan.hatter@astronomer.io
> > > > > > > > >
> > > > > > > > > > >>>>>>>>>>>>> .invalid>
> > > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at
> > > > least
> > > > > > for
> > > > > > > > > now.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to
> > > > > Airflow
> > > > > > > > > users.
> > > > > > > > > > >>>>>>>>> Indeed,
> > > > > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > > > > >>>>>>>>>>>>> has
> > > > > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some
> > > mathematicians
> > > > > and
> > > > > > > > > > >>>>>> software
> > > > > > > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> > > matter.
> > > > > For
> > > > > > > > > someone
> > > > > > > > > > >>>>>>>> new
> > > > > > > > > > >>>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow
> > orchestration,
> > > > one
> > > > > > of
> > > > > > > > the
> > > > > > > > > > >>>>>>>> first
> > > > > > > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > > > > > > >>>>>>>>>>>>>>> will
> > > > > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?"
> > > The
> > > > > > answer
> > > > > > > > is
> > > > > > > > > > >>>>>>>> almost
> > > > > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph.
> > You
> > > > > don't
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > >>>>>>>>> worry
> > > > > > > > > > >>>>>>>>>>>>>>>>> about what
> > > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your
> > workflow."
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> > least
> > > a
> > > > > > couple
> > > > > > > > > > >>>>>>>> important
> > > > > > > > > > >>>>>>>>>>>> reasons:
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned
> > > > above,
> > > > > > "DAG"
> > > > > > > > > is
> > > > > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing.
> > > We
> > > > > want
> > > > > > > > > Airflow
> > > > > > > > > > >>>>>>>> to
> > > > > > > > > > >>>>>>>>> be
> > > > > > > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > > > > > > >>>>>>>>>>>>>>> using
> > > > > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the
> > > bat
> > > > > > creates
> > > > > > > > > an
> > > > > > > > > > >>>>>>>>> initial
> > > > > > > > > > >>>>>>>>>>>>>>> barrier to
> > > > > > > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> > > > Concepts*:
> > > > > > The
> > > > > > > > DAG
> > > > > > > > > > >>>>>> is
> > > > > > > > > > >>>>>>>>> just
> > > > > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The
> > > > > > workflow
> > > > > > > > > > >>>>>> includes
> > > > > > > > > > >>>>>>>>> its
> > > > > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> > other
> > > > > > > > parameters,
> > > > > > > > > > >>>>>> and
> > > > > > > > > > >>>>>>>>> other
> > > > > > > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow
> > > > > homepage
> > > > > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > > > > > > https://airflow.apache.org/>>.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by
> > > the
> > > > > > > > community
> > > > > > > > > to
> > > > > > > > > > >>>>>>>>>>>>>>> programmatically
> > > > > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is
> > Airflow?"
> > > > docs
> > > > > > page
> > > > > > > > > > >>>>>>>>>>>>>>>>> <
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>
> > > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > > > > > >
> > https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > > >
> > > > > > > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > > > > > > >>>>>>>>>>>>> we
> > > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is
> > > > > without
> > > > > > > > using
> > > > > > > > > > >>>>>>>> "DAG."
> > > > > > > > > > >>>>>>>>>>>>>>>>> It's
> > > > > > > > > > >>>>>>>>>>>>>>> only in
> > > > > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term
> > is
> > > > > > > > introduced
> > > > > > > > > > >>>>>> out
> > > > > > > > > > >>>>>>>> of
> > > > > > > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > > > > > > >>>>>>>>>>>>>>> as a
> > > > > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain
> > it:
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> > collection
> > > > of
> > > > > > tasks
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in
> > > these
> > > > > > > > > > >>>>>> introductions
> > > > > > > > > > >>>>>>>> to
> > > > > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> > > orchestrate
> > > > > > DAGs;
> > > > > > > > it
> > > > > > > > > > >>>>>>>>>>> orchestrates
> > > > > > > > > > >>>>>>>>>>>> workflows*.
> > > > > > > > > > >>>>>>>>>>>>>>> The
> > > > > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> > > irrelevant
> > > > > to
> > > > > > > > almost
> > > > > > > > > > >>>>>>>> every
> > > > > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias
> > for
> > > > the
> > > > > > term
> > > > > > > > > "DAG"
> > > > > > > > > > >>>>>>>> and
> > > > > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG"
> > > with
> > > > > > > > > "workflow".
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running
> > away.
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>
> > > > > > ------------------------------------------------------------------
> > > > > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > > > > > > dev-help@airflow.apache.org>
> > > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>
> > > > > > -------------------------------------------------------------------
> > > > > > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > > > >>>>>> dev-help@airflow.apache.org <mailto:
> > > > > > dev-help@airflow.apache.org
> > > > > > > > >
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > > dev-help@airflow.apache.org <mailto:
> > > dev-help@airflow.apache.org>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > <mailto:
> > > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > > For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > > > <mailto:
> > > > > > > > dev-help@airflow.apache.org>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hard to say until it's looked at :)
On Mon, Feb 17, 2025 at 8:45âŻPM Aritra Basu <ar...@gmail.com>
wrote:
> I can take it up, it's mostly just a doc update right? Or are we doing code
> files replacement too?
> --
> Regards,
> Aritra Basu
>
> On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com> wrote:
>
> > Sounds like another +1000 files big PR is coming :) ? Any volunteers to
> > make it ? It's fun.
> >
> > On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com> wrote:
> >
> > > +1 for ruff rules :)
> > >
> > > Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow
> > > docs, in line with the new UI changes and to make the renaming
> > > consistent across user-facing pages.
> > >
> > > Regards,
> > > Omkar
> > >
> > > On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > +1 :) . Maybe we could add a ruff rule for that :)
> > > >
> > > > On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org> wrote:
> > > >
> > > > > It seems that our current conclusion is to use "dag" or "Dag"
> instead
> > > of
> > > > > "DAG" whenever possible. Should we replace all "DAG" in the
> codebase
> > > with
> > > > > "dag" or "Dag"? If it's too late for that (which it might be đ€),
> > > should
> > > > we
> > > > > at least avoid introducing new "DAG" in the following PRs?
> > > > >
> > > > > Best,
> > > > > Wei
> > > > >
> > > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > > > > https://github.com/apache/airflow/pull/43325
> > > > > > Let me know if that's the direction we want to go.
> > > > > >
> > > > > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > > > > <rb...@amazon.com.invalid> wrote:
> > > > > >
> > > > > > > I think Brent & Daniel summarized it best, "dag" is synonymous
> > with
> > > > > > > workflows in Airflow through the way we talk and explain what
> > > Airflow
> > > > > is
> > > > > > > all about. Although folks would ask, I donât ever use the
> > > > mathematical
> > > > > > > definition of DAG.
> > > > > > > It will be challenging and possibly confusing for many users
> > making
> > > > > such a
> > > > > > > change - I would rather direct that energy to appending the
> > Oxford
> > > > > > > definition of "dag" to include a reference to workflows in
> > Airflow.
> > > > __
> > > > > > >
> > > > > > > -- Rajesh
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> > <mailto:
> > > > > > > jarek@potiuk.com>> wrote:
> > > > > > >
> > > > > > >
> > > > > > > CAUTION: This email originated from outside of the
> organization.
> > Do
> > > > not
> > > > > > > click links or open attachments unless you can confirm the
> sender
> > > and
> > > > > know
> > > > > > > the content is safe.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> expéditeur
> > > > > externe.
> > > > > > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si
> vous
> > > ne
> > > > > pouvez
> > > > > > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> > > > certain
> > > > > que
> > > > > > > le contenu ne présente aucun risque.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From Guido's post:
> > > > > > >
> > > > > > >
> > > > > > > "Around this time the renaming seems to have been renamed".
> > > > > > >
> > > > > > >
> > > > > > > Naming is hard.
> > > > > > >
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <
> droiddev510e@gmail.com
> > > > > <mailto:
> > > > > > > droiddev510e@gmail.com>> wrote:
> > > > > > >
> > > > > > >
> > > > > > > > Yes, completely agree with above comments, dag is an Airflow
> > term
> > > > now
> > > > > > > > rather
> > > > > > > > than just a "directed acyclic graph". Believe it or not, I've
> > > > worked
> > > > > with
> > > > > > > > work
> > > > > > > > folks who've been using dags in Airflow for years now and are
> > pro
> > > > > devs
> > > > > > > but
> > > > > > > > have trouble remembering the full form of a DAG! For them,
> dag
> > =
> > > > > Airflow.
> > > > > > > >
> > > > > > > > workflow, pipeline, flow are surely better terms but will be
> a
> > > > major
> > > > > > > > behavior
> > > > > > > > change for the developer community, so we'll need a solid
> plan
> > on
> > > > > how to
> > > > > > > > introduce it, when we do.
> > > > > > > >
> > > > > > > > While we discuss this, I'd like to share about the Great
> > Renaming
> > > > in
> > > > > core
> > > > > > > > Python started by Guido back in 2009. We could probably get
> > some
> > > > > > > learnings
> > > > > > > > from there? Who knows!
> > > > > > > >
> > > > > > > > Guido's blog (2009):
> > > > > > > >
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > > <
> > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > >
> > > > > > > > Follow-up discussion (2024):
> > > > > > > >
> > https://discuss.python.org/t/finishing-the-great-renaming/54082
> > > <
> > > > > > >
> https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Omkar
> > > > > > > >
> > > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com
> > > > > <mailto:
> > > > > > > weilee.rx@gmail.com>> wrote:
> > > > > > > >
> > > > > > > > > I think we should probably just accept it as an airflow
> term.
> > > At
> > > > > least,
> > > > > > > > > thatâs how I understood it when I first used Airflow. I
> feel
> > > > > renaming
> > > > > > > it
> > > > > > > > at
> > > > > > > > > this stage would require considerable effort from
> maintainers
> > > and
> > > > > > > > existing
> > > > > > > > > users without providing equivalent benefits.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Wei
> > > > > > > > >
> > > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> > kaxilnaik@gmail.com
> > > > > > > <ma...@gmail.com>> wrote:
> > > > > > > > > >
> > > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle
> this
> > > > > discussion
> > > > > > > > for
> > > > > > > > > > Airflow 4 :) -- but right now it will cause too much
> > > disruption
> > > > > > > > > >
> > > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > > > > > <constance@astronomer.io.inva <mailto:
> > > > > constance@astronomer.io.inva>lid>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> In my experience, when you ask those with Airflow
> > experience
> > > > > what a
> > > > > > > > dag
> > > > > > > > > is,
> > > > > > > > > >> they'll start talking about workflow attributes - stuff
> > like
> > > > > dags
> > > > > > > > being
> > > > > > > > > a
> > > > > > > > > >> series of steps or tasks with owners. The structure
> > doesn't
> > > > > come up.
> > > > > > > > > >>
> > > > > > > > > >> Echo-ing others, at this point, my vote is to embrace
> the
> > > name
> > > > > and
> > > > > > > > > >> de-emphasize the mathematical structure aspect.
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > > > > <vikram@astronomer.io.inva <mailto:
> vikram@astronomer.io.inva
> > > > >lid>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> It's an interesting discussion and I remember
> struggling
> > > with
> > > > > this
> > > > > > > > > when I
> > > > > > > > > >>> started working with Airflow.
> > > > > > > > > >>> But, I also agree with the viewpoint of it being an
> > > > established
> > > > > > > > concept
> > > > > > > > > >> now
> > > > > > > > > >>> regardless of the origin.
> > > > > > > > > >>>
> > > > > > > > > >>> I am personally leaning towards the perspective best
> > > > expressed
> > > > > by
> > > > > > > > > Daniel
> > > > > > > > > >>> Standish and Brent of using Dag as a word, rather than
> > the
> > > > > computer
> > > > > > > > > >> science
> > > > > > > > > >>> concept.
> > > > > > > > > >>>
> > > > > > > > > >>> Best regards,
> > > > > > > > > >>> Vikram
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > > > > > >> <onikolas@amazon.com.inva <mailto:
> > onikolas@amazon.com.inva
> > > > >lid
> > > > > > > > > >>>>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> I agree with the general sentiment of: You're right
> > Ryan,
> > > > DAG
> > > > > > > isn't
> > > > > > > > > >> great
> > > > > > > > > >>>> and I'd rather workflow, but changing it will cause
> much
> > > > more
> > > > > > > > wreckage
> > > > > > > > > >>> than
> > > > > > > > > >>>> it solves.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Also agree with the idea to just move away from
> defining
> > > > DAG.
> > > > > I
> > > > > > > > think
> > > > > > > > > >>>> we've been naturally doing that as a community for a
> > while
> > > > now
> > > > > > > > anyway,
> > > > > > > > > >> so
> > > > > > > > > >>>> that feels like a natural step.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Cheers,
> > > > > > > > > >>>> Niko
> > > > > > > > > >>>>
> > > > > > > > > >>>> ________________________________
> > > > > > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > > > > ash@apache.org>>
> > > > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > > > > >>>> To: dev@airflow.apache.org <mailto:
> > dev@airflow.apache.org
> > > >
> > > > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term
> > "DAG"
> > > > > for end
> > > > > > > > > >> users
> > > > > > > > > >>>>
> > > > > > > > > >>>> CAUTION: This email originated from outside of the
> > > > > organization.
> > > > > > > Do
> > > > > > > > > not
> > > > > > > > > >>>> click links or open attachments unless you can confirm
> > the
> > > > > sender
> > > > > > > > and
> > > > > > > > > >>> know
> > > > > > > > > >>>> the content is safe.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > > > > expéditeur
> > > > > > > > > >> externe.
> > > > > > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> > jointe
> > > si
> > > > > vous
> > > > > > > ne
> > > > > > > > > >>> pouvez
> > > > > > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous
> > nâĂȘtes
> > > > pas
> > > > > > > > certain
> > > > > > > > > >>> que
> > > > > > > > > >>>> le contenu ne présente aucun risque.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best argument in favour of keeping âdagsâ as a term â
> > > > getting
> > > > > to
> > > > > > > > > re-use
> > > > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > > > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > > > >>>>
> > > > > > > > > >>>> In all seriousness: I donât mind either way, both
> sides
> > > have
> > > > > good
> > > > > > > > > >> reasons
> > > > > > > > > >>>> presented.
> > > > > > > > > >>>>
> > > > > > > > > >>>> -a
> > > > > > > > > >>>>
> > > > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > > > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Yeah just say, when asked where the name comes from,
> > > "well,
> > > > > no
> > > > > > > one
> > > > > > > > > >>>> actually
> > > > > > > > > >>>>> knows but..." and then make something up.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > > > > jarek@potiuk.com
> > > > > > > <ma...@potiuk.com>>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > > > > > > tongue-twister,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > > > > jarek@potiuk.com
> > > > > > > <ma...@potiuk.com>>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would
> very
> > > > much
> > > > > want
> > > > > > > > to
> > > > > > > > > >> be
> > > > > > > > > >>>>>> able
> > > > > > > > > >>>>>>> to say just "dag" without explaining it further.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > > > > > > tongue-twister,
> > > > > > > > > >>> and
> > > > > > > > > >>>> I
> > > > > > > > > >>>>>>> almost stutter on trying to recall how to pronounce
> > it
> > > > > > > properly.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> J.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > > > > > >>>>>> <brent@astronomer.io.inva <mailto:
> > > > brent@astronomer.io.inva
> > > > > >lid>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> I remember we explored renaming "DAG" when
> starting
> > on
> > > > > AIP-38
> > > > > > > to
> > > > > > > > > >>>>>> modernize
> > > > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > > > > descriptive of
> > > > > > > > what
> > > > > > > > > >>> one
> > > > > > > > > >>>>>> is
> > > > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > > > > > > implementation
> > > > > > > > > >>>> detail.
> > > > > > > > > >>>>>>>> But I agree with Daniel Standish, at this point
> > "DAG"
> > > > has
> > > > > > > become
> > > > > > > > > >>> "dag"
> > > > > > > > > >>>>>> , a
> > > > > > > > > >>>>>>>> word in its own right.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Examples for "dag" are abound in community
> > discussion,
> > > > > Airflow
> > > > > > > > > >>> Summit
> > > > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's
> > embrace
> > > > > "dag".
> > > > > > > A
> > > > > > > > > >> user
> > > > > > > > > >>>>>> just
> > > > > > > > > >>>>>>>> needs to learn one new word vs the technical
> concept
> > > > > behind
> > > > > > > that
> > > > > > > > > >>>> word. I
> > > > > > > > > >>>>>>>> think that is much less effort than refactoring so
> > > much
> > > > > code,
> > > > > > > > > >>>>>>>> documentation, blog posts, stack overflow
> questions,
> > > > etc.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > > > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > > > > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of
> > pain
> > > > for
> > > > > > > > > >>>> questionable
> > > > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I
> > > agree
> > > > > the
> > > > > > > > > >>>>>>>> association
> > > > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> And along those lines, I offer here some less
> > > invasive
> > > > > > > > > >> mitigations.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> One thing we can do no matter what is to
> > de-emphasize
> > > > the
> > > > > > > math
> > > > > > > > > >> nerd
> > > > > > > > > >>>>>>>> origins
> > > > > > > > > >>>>>>>>> of the name. That is to say, in docs / website /
> > etc,
> > > > > *never
> > > > > > > > > >>> define*
> > > > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic
> > graph.
> > > > > Always
> > > > > > > > > >> define
> > > > > > > > > >>>> it
> > > > > > > > > >>>>>>>> as a
> > > > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a
> > > > > historical
> > > > > > > > > >>>> footnote,
> > > > > > > > > >>>>>>>> and
> > > > > > > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> We could also start using lowercase in the docs
> in
> > > > > general
> > > > > > > e.g.
> > > > > > > > > >>>>>> writing
> > > > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs"
> > etc.
> > > > The
> > > > > > > > upper
> > > > > > > > > >>> case
> > > > > > > > > >>>>>>>> part
> > > > > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in
> > > > > airlfow is
> > > > > > > > just
> > > > > > > > > >>> an
> > > > > > > > > >>>>>>>>> airflow concept and the association with "DAGs"
> is
> > > not
> > > > > really
> > > > > > > > > >>>>>> unhelpful.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is
> its
> > > own
> > > > > > > thing,
> > > > > > > > is
> > > > > > > > > >>>>>>>>> *not* strictly
> > > > > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody
> > knows
> > > > > about
> > > > > > > > > >>> anyway),
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>> tell them what it is in simple terms that normal
> > > people
> > > > > > > > > >> understand.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > > > > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > > > > >
> > > > > > > > > >>>>>> wrote:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will
> be
> > > > > extremely
> > > > > > > > > >>>>>>>> difficult to
> > > > > > > > > >>>>>>>>>> get rid of it completely. Also I think it will
> > make
> > > a
> > > > > lot of
> > > > > > > > > >>>>>> "google"
> > > > > > > > > >>>>>>>>>> searches and "stack overflow" searches not
> finding
> > > the
> > > > > right
> > > > > > > > > >>>>>> answers.
> > > > > > > > > >>>>>>>>> This
> > > > > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the
> > > > > community
> > > > > > > and
> > > > > > > > > >>> ideas
> > > > > > > > > >>>>>>>> that
> > > > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of
> examples,
> > > > > problems
> > > > > > > and
> > > > > > > > > >>>>>>>> solutions
> > > > > > > > > >>>>>>>>>> you can so easily find (and we have to remember
> > that
> > > > > all the
> > > > > > > > AI
> > > > > > > > > >>>>>>>> trained
> > > > > > > > > >>>>>>>>> on
> > > > > > > > > >>>>>>>>>> past data will be also rather poorly matching
> > > queries
> > > > of
> > > > > > > > people.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily
> > switch.
> > > > > And if
> > > > > > > we
> > > > > > > > > >> do
> > > > > > > > > >>> -
> > > > > > > > > >>>>>> I
> > > > > > > > > >>>>>>>>>> would be for using workflow or pipeline instead
> of
> > > > > `dag` if
> > > > > > > > not
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>>> above
> > > > > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it
> > > might
> > > > > cause
> > > > > > > > more
> > > > > > > > > >>>>>>>> problems
> > > > > > > > > >>>>>>>>>> than it solves.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> But I am not 100% against - if others will think
> > > it's
> > > > a
> > > > > good
> > > > > > > > > >>> idea, I
> > > > > > > > > >>>>>>>> am
> > > > > > > > > >>>>>>>>> ok
> > > > > > > > > >>>>>>>>>> with it.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> J,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > > > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > > > > > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense
> to
> > > > > someone
> > > > > > > new
> > > > > > > > > >> to
> > > > > > > > > >>>>>>>>> workflow
> > > > > > > > > >>>>>>>>>>> orchestration. But it does also show the nature
> > of
> > > > > being
> > > > > > > > > >> acyclic.
> > > > > > > > > >>>>>>>> Sure,
> > > > > > > > > >>>>>>>>>> as
> > > > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it.
> > > > Still,
> > > > > in
> > > > > > > my
> > > > > > > > > >>>>>>>> opinion,
> > > > > > > > > >>>>>>>>>> there
> > > > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in
> > > workflow
> > > > > > > > > >>>>>> orchestration.
> > > > > > > > > >>>>>>>>> Most
> > > > > > > > > >>>>>>>>>>> (*if
> > > > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be
> covered
> > > > > using an
> > > > > > > > > >>> acyclic
> > > > > > > > > >>>>>>>>> manner
> > > > > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So
> I
> > > > would
> > > > > want
> > > > > > > > the
> > > > > > > > > >>>>>>>>> "acyclic"
> > > > > > > > > >>>>>>>>>>> word to stick.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Regards,
> > > > > > > > > >>>>>>>>>>> Avi
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > > > > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > > > > > > >>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an
> Airflow
> > > Fan;
> > > > > so
> > > > > > > many
> > > > > > > > > >> new
> > > > > > > > > >>>>>>>>> ideas.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be
> > replaced
> > > by
> > > > > the
> > > > > > > > more
> > > > > > > > > >>>>>>>> general
> > > > > > > > > >>>>>>>>>>> Term
> > > > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> > (Petri)
> > > > Net
> > > > > > > (AFN)
> > > > > > > > > >>>>>> (maybe
> > > > > > > > > >>>>>>>>>>> without
> > > > > > > > > >>>>>>>>>>>> a direction);
> > > > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a
> Graph
> > > > > Database.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph
> > (AFG)
> > > > > might be
> > > > > > > > > >>>>>>>> assigned
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>> an
> > > > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a
> > > > library.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> A running Graph might have a different
> structure
> > > > than
> > > > > a
> > > > > > > > > >>>>>>>> configuration
> > > > > > > > > >>>>>>>>>>>> Graph.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Best Regards
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > > > > bernd.stroehle@gmail.com>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA
> > > > > <mailto:
> > > > > > > ikholopov@google.com.INVA>LID>
> > > > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > > dev@airflow.apache.org
> > > > > > > >
> > > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term
> > > "DAG"
> > > > > for
> > > > > > > end
> > > > > > > > > >>>>>> users
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly
> > suboptimal,
> > > it
> > > > > is
> > > > > > > part
> > > > > > > > > >> of
> > > > > > > > > >>>>>>>>> Airflow
> > > > > > > > > >>>>>>>>>>>> DAG definition interface at so many levels,
> that
> > > any
> > > > > > > attempt
> > > > > > > > > >> to
> > > > > > > > > >>>>>>>>> change
> > > > > > > > > >>>>>>>>>> it
> > > > > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it.
> > The
> > > > > only
> > > > > > > > thing
> > > > > > > > > >>>>>>>> that is
> > > > > > > > > >>>>>>>>>>> worse
> > > > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when
> > > there
> > > > > are
> > > > > > > two
> > > > > > > > > >> ways
> > > > > > > > > >>>>>>>> to
> > > > > > > > > >>>>>>>>>>> define
> > > > > > > > > >>>>>>>>>>>> the same thing. Countless articles and
> tutorials
> > > > will
> > > > > > > > suddenly
> > > > > > > > > >>>>>>>> become
> > > > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as
> > > "DAG"s.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> We are already at risk of scaring the users
> away
> > > > with
> > > > > a
> > > > > > > > number
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>>>> breaking
> > > > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more
> > breaking
> > > > > changes
> > > > > > > > for
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>> most
> > > > > > > > > >>>>>>>>>>>> basic things is not something that people are
> > > > looking
> > > > > for.
> > > > > > > > > >>>>>>>> Attempting
> > > > > > > > > >>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>> change the fundamental terms will be
> interpreted
> > > as
> > > > an
> > > > > > > even
> > > > > > > > > >>>>>>>> stronger
> > > > > > > > > >>>>>>>>>>> signal
> > > > > > > > > >>>>>>>>>>>> of project immaturity.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the
> > term
> > > > in
> > > > > the
> > > > > > > > long
> > > > > > > > > >>>>>>>> run. I
> > > > > > > > > >>>>>>>>>>> even
> > > > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in
> > the
> > > > DAG
> > > > > > > > > >> definition
> > > > > > > > > >>>>>>>>>>> interface.
> > > > > > > > > >>>>>>>>>>>> We better put our time and efforts in other
> > places
> > > > in
> > > > > > > > Airflow,
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>> which
> > > > > > > > > >>>>>>>>>>>> there are plenty.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Kind regards,
> > > > > > > > > >>>>>>>>>>>> Igor
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > > > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > > > bas@astronomer.io.inva
> > > > > >lid
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have
> > > already
> > > > > faded
> > > > > > > > over
> > > > > > > > > >>>>>>>> time,
> > > > > > > > > >>>>>>>>>>>>> for example there are now several ways to
> > create
> > > > > cyclic
> > > > > > > > > >>>>>> graphs,
> > > > > > > > > >>>>>>>>> e.g.
> > > > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine
> these
> > > > > > > properties
> > > > > > > > > >>>>>>>>> vanishing
> > > > > > > > > >>>>>>>>>>>>> even more in the future, so from that
> > > perspective I
> > > > > > > support
> > > > > > > > > >>>>>>>>> changing
> > > > > > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do
> > naming:
> > > > > > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I
> > > suggest
> > > > > > > > defining
> > > > > > > > > >> a
> > > > > > > > > >>>>>>>>> clear
> > > > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the
> > > > > deprecation in
> > > > > > > > > >>>>>>>> Airflow
> > > > > > > > > >>>>>>>>> 3,
> > > > > > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Bas
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > > > neil4real@gmail.com
> > > > > > > <ma...@gmail.com>>
> > > > > > > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > > > especially
> > > > > when
> > > > > > > > > >>>>>> most
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or
> confusing
> > > > > about it
> > > > > > > at
> > > > > > > > > >>>>>>>> all,
> > > > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly
> > > confusing
> > > > > to
> > > > > > > most
> > > > > > > > > >>>>>>>> users
> > > > > > > > > >>>>>>>>>> who
> > > > > > > > > >>>>>>>>>>>>>> have been
> > > > > > > > > >>>>>>>>>>>>> using
> > > > > > > > > >>>>>>>>>>>>>> the term for years.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping
> Chung
> > > > > > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > > > > tp@astronomer.io.inva
> > > > > > > >lid
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the
> term
> > > > DAG.
> > > > > The
> > > > > > > > only
> > > > > > > > > >>>>>>>>>> problem
> > > > > > > > > >>>>>>>>>>>>> (aside
> > > > > > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding
> > > myselfâto
> > > > > stop
> > > > > > > > using
> > > > > > > > > >>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> term)
> > > > > > > > > >>>>>>>>>>>>> is to
> > > > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned
> > > > > âworkflowâ is
> > > > > > > > not
> > > > > > > > > >>>>>>>> very
> > > > > > > > > >>>>>>>>>>>>> accurate
> > > > > > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> > > proposed,
> > > > > but
> > > > > > > > itâs a
> > > > > > > > > >>>>>>>> bit
> > > > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places
> and
> > > kept
> > > > > > > finding
> > > > > > > > > >>>>>>>> myself
> > > > > > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to
> > > > clarify
> > > > > âDAG
> > > > > > > > > >>>>>>>>>> definitionâ
> > > > > > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> TP
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > > > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > > > > > j_scheffler@gmx.de.INVA>LID>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly
> same
> > > > > > > > observation,
> > > > > > > > > >>>>>>>> had a
> > > > > > > > > >>>>>>>>>>>>>>>> short
> > > > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always
> an
> > > > > > > introduction
> > > > > > > > > >>>>>> if
> > > > > > > > > >>>>>>>>>>>>>>> someone
> > > > > > > > > >>>>>>>>>>>>> joins
> > > > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes
> > > sense.
> > > > > > > > > >>>>>> Especially
> > > > > > > > > >>>>>>>>> when
> > > > > > > > > >>>>>>>>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> > > reasonable
> > > > > step.
> > > > > > > > > >>>>>>>> Concepts
> > > > > > > > > >>>>>>>>>>>>>>> still
> > > > > > > > > >>>>>>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding
> below
> > > the
> > > > > desk
> > > > > > > > with
> > > > > > > > > >>>>>>>> you
> > > > > > > > > >>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>> +1
> > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep
> > > > details
> > > > > of
> > > > > > > > > >>>>>> python
> > > > > > > > > >>>>>>>>> names
> > > > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAGâŠ
> > but
> > > > user
> > > > > > > > facing
> > > > > > > > > >>>>>> it
> > > > > > > > > >>>>>>>>> is a
> > > > > > > > > >>>>>>>>>>>>> workflow.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Jens
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > > > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > > > > ryan.hatter@astronomer.io
> > > > > > > >
> > > > > > > > > >>>>>>>>>>>>> .invalid>
> > > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at
> > > least
> > > > > for
> > > > > > > > now.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to
> > > > Airflow
> > > > > > > > users.
> > > > > > > > > >>>>>>>>> Indeed,
> > > > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > > > >>>>>>>>>>>>> has
> > > > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some
> > mathematicians
> > > > and
> > > > > > > > > >>>>>> software
> > > > > > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> > matter.
> > > > For
> > > > > > > > someone
> > > > > > > > > >>>>>>>> new
> > > > > > > > > >>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow
> orchestration,
> > > one
> > > > > of
> > > > > > > the
> > > > > > > > > >>>>>>>> first
> > > > > > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > > > > > >>>>>>>>>>>>>>> will
> > > > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?"
> > The
> > > > > answer
> > > > > > > is
> > > > > > > > > >>>>>>>> almost
> > > > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph.
> You
> > > > don't
> > > > > > > need
> > > > > > > > to
> > > > > > > > > >>>>>>>>> worry
> > > > > > > > > >>>>>>>>>>>>>>>>> about what
> > > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your
> workflow."
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at
> least
> > a
> > > > > couple
> > > > > > > > > >>>>>>>> important
> > > > > > > > > >>>>>>>>>>>> reasons:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned
> > > above,
> > > > > "DAG"
> > > > > > > > is
> > > > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing.
> > We
> > > > want
> > > > > > > > Airflow
> > > > > > > > > >>>>>>>> to
> > > > > > > > > >>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > > > > > >>>>>>>>>>>>>>> using
> > > > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the
> > bat
> > > > > creates
> > > > > > > > an
> > > > > > > > > >>>>>>>>> initial
> > > > > > > > > >>>>>>>>>>>>>>> barrier to
> > > > > > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> > > Concepts*:
> > > > > The
> > > > > > > DAG
> > > > > > > > > >>>>>> is
> > > > > > > > > >>>>>>>>> just
> > > > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The
> > > > > workflow
> > > > > > > > > >>>>>> includes
> > > > > > > > > >>>>>>>>> its
> > > > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen
> other
> > > > > > > parameters,
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow
> > > > homepage
> > > > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > > > > > https://airflow.apache.org/>>.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by
> > the
> > > > > > > community
> > > > > > > > to
> > > > > > > > > >>>>>>>>>>>>>>> programmatically
> > > > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is
> Airflow?"
> > > docs
> > > > > page
> > > > > > > > > >>>>>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>
> > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > > > > >
> https://airflow.apache.org/docs/apache-airflow/stable/index.html
> > >
> > > > > > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > > > > > >>>>>>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is
> > > > without
> > > > > > > using
> > > > > > > > > >>>>>>>> "DAG."
> > > > > > > > > >>>>>>>>>>>>>>>>> It's
> > > > > > > > > >>>>>>>>>>>>>>> only in
> > > > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term
> is
> > > > > > > introduced
> > > > > > > > > >>>>>> out
> > > > > > > > > >>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > > > > > >>>>>>>>>>>>>>> as a
> > > > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain
> it:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a
> collection
> > > of
> > > > > tasks
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in
> > these
> > > > > > > > > >>>>>> introductions
> > > > > > > > > >>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> > orchestrate
> > > > > DAGs;
> > > > > > > it
> > > > > > > > > >>>>>>>>>>> orchestrates
> > > > > > > > > >>>>>>>>>>>> workflows*.
> > > > > > > > > >>>>>>>>>>>>>>> The
> > > > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> > irrelevant
> > > > to
> > > > > > > almost
> > > > > > > > > >>>>>>>> every
> > > > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias
> for
> > > the
> > > > > term
> > > > > > > > "DAG"
> > > > > > > > > >>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG"
> > with
> > > > > > > > "workflow".
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running
> away.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>
> > > > > ------------------------------------------------------------------
> > > > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > > > > > dev-help@airflow.apache.org>
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>
> > > > > -------------------------------------------------------------------
> > > > > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > > >>>>>> dev-help@airflow.apache.org <mailto:
> > > > > dev-help@airflow.apache.org
> > > > > > > >
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > <mailto:
> > > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > <mailto:
> > > > > > > dev-help@airflow.apache.org>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Aritra Basu <ar...@gmail.com>.
I can take it up, it's mostly just a doc update right? Or are we doing code
files replacement too?
--
Regards,
Aritra Basu
On Tue, 18 Feb 2025, 12:55âŻam Jarek Potiuk, <ja...@potiuk.com> wrote:
> Sounds like another +1000 files big PR is coming :) ? Any volunteers to
> make it ? It's fun.
>
> On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com> wrote:
>
> > +1 for ruff rules :)
> >
> > Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow
> > docs, in line with the new UI changes and to make the renaming
> > consistent across user-facing pages.
> >
> > Regards,
> > Omkar
> >
> > On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > +1 :) . Maybe we could add a ruff rule for that :)
> > >
> > > On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org> wrote:
> > >
> > > > It seems that our current conclusion is to use "dag" or "Dag" instead
> > of
> > > > "DAG" whenever possible. Should we replace all "DAG" in the codebase
> > with
> > > > "dag" or "Dag"? If it's too late for that (which it might be đ€),
> > should
> > > we
> > > > at least avoid introducing new "DAG" in the following PRs?
> > > >
> > > > Best,
> > > > Wei
> > > >
> > > > On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > > > https://github.com/apache/airflow/pull/43325
> > > > > Let me know if that's the direction we want to go.
> > > > >
> > > > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > > > <rb...@amazon.com.invalid> wrote:
> > > > >
> > > > > > I think Brent & Daniel summarized it best, "dag" is synonymous
> with
> > > > > > workflows in Airflow through the way we talk and explain what
> > Airflow
> > > > is
> > > > > > all about. Although folks would ask, I donât ever use the
> > > mathematical
> > > > > > definition of DAG.
> > > > > > It will be challenging and possibly confusing for many users
> making
> > > > such a
> > > > > > change - I would rather direct that energy to appending the
> Oxford
> > > > > > definition of "dag" to include a reference to workflows in
> Airflow.
> > > __
> > > > > >
> > > > > > -- Rajesh
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com
> <mailto:
> > > > > > jarek@potiuk.com>> wrote:
> > > > > >
> > > > > >
> > > > > > CAUTION: This email originated from outside of the organization.
> Do
> > > not
> > > > > > click links or open attachments unless you can confirm the sender
> > and
> > > > know
> > > > > > the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
> > > > externe.
> > > > > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous
> > ne
> > > > pouvez
> > > > > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> > > certain
> > > > que
> > > > > > le contenu ne présente aucun risque.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > From Guido's post:
> > > > > >
> > > > > >
> > > > > > "Around this time the renaming seems to have been renamed".
> > > > > >
> > > > > >
> > > > > > Naming is hard.
> > > > > >
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev510e@gmail.com
> > > > <mailto:
> > > > > > droiddev510e@gmail.com>> wrote:
> > > > > >
> > > > > >
> > > > > > > Yes, completely agree with above comments, dag is an Airflow
> term
> > > now
> > > > > > > rather
> > > > > > > than just a "directed acyclic graph". Believe it or not, I've
> > > worked
> > > > with
> > > > > > > work
> > > > > > > folks who've been using dags in Airflow for years now and are
> pro
> > > > devs
> > > > > > but
> > > > > > > have trouble remembering the full form of a DAG! For them, dag
> =
> > > > Airflow.
> > > > > > >
> > > > > > > workflow, pipeline, flow are surely better terms but will be a
> > > major
> > > > > > > behavior
> > > > > > > change for the developer community, so we'll need a solid plan
> on
> > > > how to
> > > > > > > introduce it, when we do.
> > > > > > >
> > > > > > > While we discuss this, I'd like to share about the Great
> Renaming
> > > in
> > > > core
> > > > > > > Python started by Guido back in 2009. We could probably get
> some
> > > > > > learnings
> > > > > > > from there? Who knows!
> > > > > > >
> > > > > > > Guido's blog (2009):
> > > > > > >
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > > <
> > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > >
> > > > > > > Follow-up discussion (2024):
> > > > > > >
> https://discuss.python.org/t/finishing-the-great-renaming/54082
> > <
> > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Omkar
> > > > > > >
> > > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com
> > > > <mailto:
> > > > > > weilee.rx@gmail.com>> wrote:
> > > > > > >
> > > > > > > > I think we should probably just accept it as an airflow term.
> > At
> > > > least,
> > > > > > > > thatâs how I understood it when I first used Airflow. I feel
> > > > renaming
> > > > > > it
> > > > > > > at
> > > > > > > > this stage would require considerable effort from maintainers
> > and
> > > > > > > existing
> > > > > > > > users without providing equivalent benefits.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Wei
> > > > > > > >
> > > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <
> kaxilnaik@gmail.com
> > > > > > <ma...@gmail.com>> wrote:
> > > > > > > > >
> > > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this
> > > > discussion
> > > > > > > for
> > > > > > > > > Airflow 4 :) -- but right now it will cause too much
> > disruption
> > > > > > > > >
> > > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > > > > <constance@astronomer.io.inva <mailto:
> > > > constance@astronomer.io.inva>lid>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> In my experience, when you ask those with Airflow
> experience
> > > > what a
> > > > > > > dag
> > > > > > > > is,
> > > > > > > > >> they'll start talking about workflow attributes - stuff
> like
> > > > dags
> > > > > > > being
> > > > > > > > a
> > > > > > > > >> series of steps or tasks with owners. The structure
> doesn't
> > > > come up.
> > > > > > > > >>
> > > > > > > > >> Echo-ing others, at this point, my vote is to embrace the
> > name
> > > > and
> > > > > > > > >> de-emphasize the mathematical structure aspect.
> > > > > > > > >>
> > > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > > > <vikram@astronomer.io.inva <mailto:vikram@astronomer.io.inva
> > > >lid>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> It's an interesting discussion and I remember struggling
> > with
> > > > this
> > > > > > > > when I
> > > > > > > > >>> started working with Airflow.
> > > > > > > > >>> But, I also agree with the viewpoint of it being an
> > > established
> > > > > > > concept
> > > > > > > > >> now
> > > > > > > > >>> regardless of the origin.
> > > > > > > > >>>
> > > > > > > > >>> I am personally leaning towards the perspective best
> > > expressed
> > > > by
> > > > > > > > Daniel
> > > > > > > > >>> Standish and Brent of using Dag as a word, rather than
> the
> > > > computer
> > > > > > > > >> science
> > > > > > > > >>> concept.
> > > > > > > > >>>
> > > > > > > > >>> Best regards,
> > > > > > > > >>> Vikram
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > > > > >> <onikolas@amazon.com.inva <mailto:
> onikolas@amazon.com.inva
> > > >lid
> > > > > > > > >>>>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> I agree with the general sentiment of: You're right
> Ryan,
> > > DAG
> > > > > > isn't
> > > > > > > > >> great
> > > > > > > > >>>> and I'd rather workflow, but changing it will cause much
> > > more
> > > > > > > wreckage
> > > > > > > > >>> than
> > > > > > > > >>>> it solves.
> > > > > > > > >>>>
> > > > > > > > >>>> Also agree with the idea to just move away from defining
> > > DAG.
> > > > I
> > > > > > > think
> > > > > > > > >>>> we've been naturally doing that as a community for a
> while
> > > now
> > > > > > > anyway,
> > > > > > > > >> so
> > > > > > > > >>>> that feels like a natural step.
> > > > > > > > >>>>
> > > > > > > > >>>> Cheers,
> > > > > > > > >>>> Niko
> > > > > > > > >>>>
> > > > > > > > >>>> ________________________________
> > > > > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > > > ash@apache.org>>
> > > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > > > >>>> To: dev@airflow.apache.org <mailto:
> dev@airflow.apache.org
> > >
> > > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term
> "DAG"
> > > > for end
> > > > > > > > >> users
> > > > > > > > >>>>
> > > > > > > > >>>> CAUTION: This email originated from outside of the
> > > > organization.
> > > > > > Do
> > > > > > > > not
> > > > > > > > >>>> click links or open attachments unless you can confirm
> the
> > > > sender
> > > > > > > and
> > > > > > > > >>> know
> > > > > > > > >>>> the content is safe.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > > > expéditeur
> > > > > > > > >> externe.
> > > > > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce
> jointe
> > si
> > > > vous
> > > > > > ne
> > > > > > > > >>> pouvez
> > > > > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous
> nâĂȘtes
> > > pas
> > > > > > > certain
> > > > > > > > >>> que
> > > > > > > > >>>> le contenu ne présente aucun risque.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Best argument in favour of keeping âdagsâ as a term â
> > > getting
> > > > to
> > > > > > > > re-use
> > > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > > >>>>
> > > > > > > > >>>> In all seriousness: I donât mind either way, both sides
> > have
> > > > good
> > > > > > > > >> reasons
> > > > > > > > >>>> presented.
> > > > > > > > >>>>
> > > > > > > > >>>> -a
> > > > > > > > >>>>
> > > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>> Yeah just say, when asked where the name comes from,
> > "well,
> > > > no
> > > > > > one
> > > > > > > > >>>> actually
> > > > > > > > >>>>> knows but..." and then make something up.
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > > > jarek@potiuk.com
> > > > > > <ma...@potiuk.com>>
> > > > > > > > >> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > > > > > tongue-twister,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > > > jarek@potiuk.com
> > > > > > <ma...@potiuk.com>>
> > > > > > > > >>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very
> > > much
> > > > want
> > > > > > > to
> > > > > > > > >> be
> > > > > > > > >>>>>> able
> > > > > > > > >>>>>>> to say just "dag" without explaining it further.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > > > > > tongue-twister,
> > > > > > > > >>> and
> > > > > > > > >>>> I
> > > > > > > > >>>>>>> almost stutter on trying to recall how to pronounce
> it
> > > > > > properly.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> J.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > > > > >>>>>> <brent@astronomer.io.inva <mailto:
> > > brent@astronomer.io.inva
> > > > >lid>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> I remember we explored renaming "DAG" when starting
> on
> > > > AIP-38
> > > > > > to
> > > > > > > > >>>>>> modernize
> > > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > > > descriptive of
> > > > > > > what
> > > > > > > > >>> one
> > > > > > > > >>>>>> is
> > > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > > > > > implementation
> > > > > > > > >>>> detail.
> > > > > > > > >>>>>>>> But I agree with Daniel Standish, at this point
> "DAG"
> > > has
> > > > > > become
> > > > > > > > >>> "dag"
> > > > > > > > >>>>>> , a
> > > > > > > > >>>>>>>> word in its own right.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Examples for "dag" are abound in community
> discussion,
> > > > Airflow
> > > > > > > > >>> Summit
> > > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's
> embrace
> > > > "dag".
> > > > > > A
> > > > > > > > >> user
> > > > > > > > >>>>>> just
> > > > > > > > >>>>>>>> needs to learn one new word vs the technical concept
> > > > behind
> > > > > > that
> > > > > > > > >>>> word. I
> > > > > > > > >>>>>>>> think that is much less effort than refactoring so
> > much
> > > > code,
> > > > > > > > >>>>>>>> documentation, blog posts, stack overflow questions,
> > > etc.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > > > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of
> pain
> > > for
> > > > > > > > >>>> questionable
> > > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I
> > agree
> > > > the
> > > > > > > > >>>>>>>> association
> > > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> And along those lines, I offer here some less
> > invasive
> > > > > > > > >> mitigations.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> One thing we can do no matter what is to
> de-emphasize
> > > the
> > > > > > math
> > > > > > > > >> nerd
> > > > > > > > >>>>>>>> origins
> > > > > > > > >>>>>>>>> of the name. That is to say, in docs / website /
> etc,
> > > > *never
> > > > > > > > >>> define*
> > > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic
> graph.
> > > > Always
> > > > > > > > >> define
> > > > > > > > >>>> it
> > > > > > > > >>>>>>>> as a
> > > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a
> > > > historical
> > > > > > > > >>>> footnote,
> > > > > > > > >>>>>>>> and
> > > > > > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> We could also start using lowercase in the docs in
> > > > general
> > > > > > e.g.
> > > > > > > > >>>>>> writing
> > > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs"
> etc.
> > > The
> > > > > > > upper
> > > > > > > > >>> case
> > > > > > > > >>>>>>>> part
> > > > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in
> > > > airlfow is
> > > > > > > just
> > > > > > > > >>> an
> > > > > > > > >>>>>>>>> airflow concept and the association with "DAGs" is
> > not
> > > > really
> > > > > > > > >>>>>> unhelpful.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its
> > own
> > > > > > thing,
> > > > > > > is
> > > > > > > > >>>>>>>>> *not* strictly
> > > > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody
> knows
> > > > about
> > > > > > > > >>> anyway),
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>>>> tell them what it is in simple terms that normal
> > people
> > > > > > > > >> understand.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > > > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > > > >
> > > > > > > > >>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be
> > > > extremely
> > > > > > > > >>>>>>>> difficult to
> > > > > > > > >>>>>>>>>> get rid of it completely. Also I think it will
> make
> > a
> > > > lot of
> > > > > > > > >>>>>> "google"
> > > > > > > > >>>>>>>>>> searches and "stack overflow" searches not finding
> > the
> > > > right
> > > > > > > > >>>>>> answers.
> > > > > > > > >>>>>>>>> This
> > > > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the
> > > > community
> > > > > > and
> > > > > > > > >>> ideas
> > > > > > > > >>>>>>>> that
> > > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples,
> > > > problems
> > > > > > and
> > > > > > > > >>>>>>>> solutions
> > > > > > > > >>>>>>>>>> you can so easily find (and we have to remember
> that
> > > > all the
> > > > > > > AI
> > > > > > > > >>>>>>>> trained
> > > > > > > > >>>>>>>>> on
> > > > > > > > >>>>>>>>>> past data will be also rather poorly matching
> > queries
> > > of
> > > > > > > people.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily
> switch.
> > > > And if
> > > > > > we
> > > > > > > > >> do
> > > > > > > > >>> -
> > > > > > > > >>>>>> I
> > > > > > > > >>>>>>>>>> would be for using workflow or pipeline instead of
> > > > `dag` if
> > > > > > > not
> > > > > > > > >>> the
> > > > > > > > >>>>>>>> above
> > > > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it
> > might
> > > > cause
> > > > > > > more
> > > > > > > > >>>>>>>> problems
> > > > > > > > >>>>>>>>>> than it solves.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> But I am not 100% against - if others will think
> > it's
> > > a
> > > > good
> > > > > > > > >>> idea, I
> > > > > > > > >>>>>>>> am
> > > > > > > > >>>>>>>>> ok
> > > > > > > > >>>>>>>>>> with it.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> J,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > > > > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to
> > > > someone
> > > > > > new
> > > > > > > > >> to
> > > > > > > > >>>>>>>>> workflow
> > > > > > > > >>>>>>>>>>> orchestration. But it does also show the nature
> of
> > > > being
> > > > > > > > >> acyclic.
> > > > > > > > >>>>>>>> Sure,
> > > > > > > > >>>>>>>>>> as
> > > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it.
> > > Still,
> > > > in
> > > > > > my
> > > > > > > > >>>>>>>> opinion,
> > > > > > > > >>>>>>>>>> there
> > > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in
> > workflow
> > > > > > > > >>>>>> orchestration.
> > > > > > > > >>>>>>>>> Most
> > > > > > > > >>>>>>>>>>> (*if
> > > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered
> > > > using an
> > > > > > > > >>> acyclic
> > > > > > > > >>>>>>>>> manner
> > > > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I
> > > would
> > > > want
> > > > > > > the
> > > > > > > > >>>>>>>>> "acyclic"
> > > > > > > > >>>>>>>>>>> word to stick.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Regards,
> > > > > > > > >>>>>>>>>>> Avi
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > > > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > > > > > >>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow
> > Fan;
> > > > so
> > > > > > many
> > > > > > > > >> new
> > > > > > > > >>>>>>>>> ideas.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be
> replaced
> > by
> > > > the
> > > > > > > more
> > > > > > > > >>>>>>>> general
> > > > > > > > >>>>>>>>>>> Term
> > > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow
> (Petri)
> > > Net
> > > > > > (AFN)
> > > > > > > > >>>>>> (maybe
> > > > > > > > >>>>>>>>>>> without
> > > > > > > > >>>>>>>>>>>> a direction);
> > > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph
> > > > Database.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph
> (AFG)
> > > > might be
> > > > > > > > >>>>>>>> assigned
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>> an
> > > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a
> > > library.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> A running Graph might have a different structure
> > > than
> > > > a
> > > > > > > > >>>>>>>> configuration
> > > > > > > > >>>>>>>>>>>> Graph.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Best Regards
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > > > bernd.stroehle@gmail.com>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA
> > > > <mailto:
> > > > > > ikholopov@google.com.INVA>LID>
> > > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > > dev@airflow.apache.org
> > > > > > >
> > > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term
> > "DAG"
> > > > for
> > > > > > end
> > > > > > > > >>>>>> users
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly
> suboptimal,
> > it
> > > > is
> > > > > > part
> > > > > > > > >> of
> > > > > > > > >>>>>>>>> Airflow
> > > > > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that
> > any
> > > > > > attempt
> > > > > > > > >> to
> > > > > > > > >>>>>>>>> change
> > > > > > > > >>>>>>>>>> it
> > > > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it.
> The
> > > > only
> > > > > > > thing
> > > > > > > > >>>>>>>> that is
> > > > > > > > >>>>>>>>>>> worse
> > > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when
> > there
> > > > are
> > > > > > two
> > > > > > > > >> ways
> > > > > > > > >>>>>>>> to
> > > > > > > > >>>>>>>>>>> define
> > > > > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials
> > > will
> > > > > > > suddenly
> > > > > > > > >>>>>>>> become
> > > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as
> > "DAG"s.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> We are already at risk of scaring the users away
> > > with
> > > > a
> > > > > > > number
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>>>> breaking
> > > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more
> breaking
> > > > changes
> > > > > > > for
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>> most
> > > > > > > > >>>>>>>>>>>> basic things is not something that people are
> > > looking
> > > > for.
> > > > > > > > >>>>>>>> Attempting
> > > > > > > > >>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted
> > as
> > > an
> > > > > > even
> > > > > > > > >>>>>>>> stronger
> > > > > > > > >>>>>>>>>>> signal
> > > > > > > > >>>>>>>>>>>> of project immaturity.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the
> term
> > > in
> > > > the
> > > > > > > long
> > > > > > > > >>>>>>>> run. I
> > > > > > > > >>>>>>>>>>> even
> > > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in
> the
> > > DAG
> > > > > > > > >> definition
> > > > > > > > >>>>>>>>>>> interface.
> > > > > > > > >>>>>>>>>>>> We better put our time and efforts in other
> places
> > > in
> > > > > > > Airflow,
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>> which
> > > > > > > > >>>>>>>>>>>> there are plenty.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Kind regards,
> > > > > > > > >>>>>>>>>>>> Igor
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > > bas@astronomer.io.inva
> > > > >lid
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have
> > already
> > > > faded
> > > > > > > over
> > > > > > > > >>>>>>>> time,
> > > > > > > > >>>>>>>>>>>>> for example there are now several ways to
> create
> > > > cyclic
> > > > > > > > >>>>>> graphs,
> > > > > > > > >>>>>>>>> e.g.
> > > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these
> > > > > > properties
> > > > > > > > >>>>>>>>> vanishing
> > > > > > > > >>>>>>>>>>>>> even more in the future, so from that
> > perspective I
> > > > > > support
> > > > > > > > >>>>>>>>> changing
> > > > > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do
> naming:
> > > > > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I
> > suggest
> > > > > > > defining
> > > > > > > > >> a
> > > > > > > > >>>>>>>>> clear
> > > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the
> > > > deprecation in
> > > > > > > > >>>>>>>> Airflow
> > > > > > > > >>>>>>>>> 3,
> > > > > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Bas
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > > neil4real@gmail.com
> > > > > > <ma...@gmail.com>>
> > > > > > > wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > > especially
> > > > when
> > > > > > > > >>>>>> most
> > > > > > > > >>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing
> > > > about it
> > > > > > at
> > > > > > > > >>>>>>>> all,
> > > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly
> > confusing
> > > > to
> > > > > > most
> > > > > > > > >>>>>>>> users
> > > > > > > > >>>>>>>>>> who
> > > > > > > > >>>>>>>>>>>>>> have been
> > > > > > > > >>>>>>>>>>>>> using
> > > > > > > > >>>>>>>>>>>>>> the term for years.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung
> > > > > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > > > tp@astronomer.io.inva
> > > > > > >lid
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term
> > > DAG.
> > > > The
> > > > > > > only
> > > > > > > > >>>>>>>>>> problem
> > > > > > > > >>>>>>>>>>>>> (aside
> > > > > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding
> > myselfâto
> > > > stop
> > > > > > > using
> > > > > > > > >>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> term)
> > > > > > > > >>>>>>>>>>>>> is to
> > > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned
> > > > âworkflowâ is
> > > > > > > not
> > > > > > > > >>>>>>>> very
> > > > > > > > >>>>>>>>>>>>> accurate
> > > > > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> > proposed,
> > > > but
> > > > > > > itâs a
> > > > > > > > >>>>>>>> bit
> > > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and
> > kept
> > > > > > finding
> > > > > > > > >>>>>>>> myself
> > > > > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to
> > > clarify
> > > > âDAG
> > > > > > > > >>>>>>>>>> definitionâ
> > > > > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> TP
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > > > > j_scheffler@gmx.de.INVA>LID>
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same
> > > > > > > observation,
> > > > > > > > >>>>>>>> had a
> > > > > > > > >>>>>>>>>>>>>>>> short
> > > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an
> > > > > > introduction
> > > > > > > > >>>>>> if
> > > > > > > > >>>>>>>>>>>>>>> someone
> > > > > > > > >>>>>>>>>>>>> joins
> > > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes
> > sense.
> > > > > > > > >>>>>> Especially
> > > > > > > > >>>>>>>>> when
> > > > > > > > >>>>>>>>>>>>>>> we
> > > > > > > > >>>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> > reasonable
> > > > step.
> > > > > > > > >>>>>>>> Concepts
> > > > > > > > >>>>>>>>>>>>>>> still
> > > > > > > > >>>>>>>>>>>>> can
> > > > > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding below
> > the
> > > > desk
> > > > > > > with
> > > > > > > > >>>>>>>> you
> > > > > > > > >>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>> +1
> > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep
> > > details
> > > > of
> > > > > > > > >>>>>> python
> > > > > > > > >>>>>>>>> names
> > > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAGâŠ
> but
> > > user
> > > > > > > facing
> > > > > > > > >>>>>> it
> > > > > > > > >>>>>>>>> is a
> > > > > > > > >>>>>>>>>>>>> workflow.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Jens
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > > > ryan.hatter@astronomer.io
> > > > > > >
> > > > > > > > >>>>>>>>>>>>> .invalid>
> > > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at
> > least
> > > > for
> > > > > > > now.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to
> > > Airflow
> > > > > > > users.
> > > > > > > > >>>>>>>>> Indeed,
> > > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > > >>>>>>>>>>>>> has
> > > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some
> mathematicians
> > > and
> > > > > > > > >>>>>> software
> > > > > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually
> matter.
> > > For
> > > > > > > someone
> > > > > > > > >>>>>>>> new
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration,
> > one
> > > > of
> > > > > > the
> > > > > > > > >>>>>>>> first
> > > > > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > > > > >>>>>>>>>>>>>>> will
> > > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?"
> The
> > > > answer
> > > > > > is
> > > > > > > > >>>>>>>> almost
> > > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You
> > > don't
> > > > > > need
> > > > > > > to
> > > > > > > > >>>>>>>>> worry
> > > > > > > > >>>>>>>>>>>>>>>>> about what
> > > > > > > > >>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow."
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least
> a
> > > > couple
> > > > > > > > >>>>>>>> important
> > > > > > > > >>>>>>>>>>>> reasons:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned
> > above,
> > > > "DAG"
> > > > > > > is
> > > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing.
> We
> > > want
> > > > > > > Airflow
> > > > > > > > >>>>>>>> to
> > > > > > > > >>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > > > > >>>>>>>>>>>>>>> using
> > > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the
> bat
> > > > creates
> > > > > > > an
> > > > > > > > >>>>>>>>> initial
> > > > > > > > >>>>>>>>>>>>>>> barrier to
> > > > > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> > Concepts*:
> > > > The
> > > > > > DAG
> > > > > > > > >>>>>> is
> > > > > > > > >>>>>>>>> just
> > > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The
> > > > workflow
> > > > > > > > >>>>>> includes
> > > > > > > > >>>>>>>>> its
> > > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other
> > > > > > parameters,
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > > > > >>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow
> > > homepage
> > > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > > > > https://airflow.apache.org/>>.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by
> the
> > > > > > community
> > > > > > > to
> > > > > > > > >>>>>>>>>>>>>>> programmatically
> > > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?"
> > docs
> > > > page
> > > > > > > > >>>>>>>>>>>>>>>>> <
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>
> > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html
> >
> > > > > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > > > > >>>>>>>>>>>>> we
> > > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is
> > > without
> > > > > > using
> > > > > > > > >>>>>>>> "DAG."
> > > > > > > > >>>>>>>>>>>>>>>>> It's
> > > > > > > > >>>>>>>>>>>>>>> only in
> > > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is
> > > > > > introduced
> > > > > > > > >>>>>> out
> > > > > > > > >>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > > > > >>>>>>>>>>>>>>> as a
> > > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection
> > of
> > > > tasks
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in
> these
> > > > > > > > >>>>>> introductions
> > > > > > > > >>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't
> orchestrate
> > > > DAGs;
> > > > > > it
> > > > > > > > >>>>>>>>>>> orchestrates
> > > > > > > > >>>>>>>>>>>> workflows*.
> > > > > > > > >>>>>>>>>>>>>>> The
> > > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons
> irrelevant
> > > to
> > > > > > almost
> > > > > > > > >>>>>>>> every
> > > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for
> > the
> > > > term
> > > > > > > "DAG"
> > > > > > > > >>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG"
> with
> > > > > > > "workflow".
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>
> > > > ------------------------------------------------------------------
> > > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > > > > dev-help@airflow.apache.org>
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>
> > > > -------------------------------------------------------------------
> > > > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > > >>>>>> dev-help@airflow.apache.org <mailto:
> > > > dev-help@airflow.apache.org
> > > > > > >
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > <mailto:
> > > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > <mailto:
> > > > > > dev-help@airflow.apache.org>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
Sounds like another +1000 files big PR is coming :) ? Any volunteers to
make it ? It's fun.
On Mon, Feb 17, 2025 at 6:52âŻPM Omkar P <dr...@gmail.com> wrote:
> +1 for ruff rules :)
>
> Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow
> docs, in line with the new UI changes and to make the renaming
> consistent across user-facing pages.
>
> Regards,
> Omkar
>
> On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > +1 :) . Maybe we could add a ruff rule for that :)
> >
> > On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org> wrote:
> >
> > > It seems that our current conclusion is to use "dag" or "Dag" instead
> of
> > > "DAG" whenever possible. Should we replace all "DAG" in the codebase
> with
> > > "dag" or "Dag"? If it's too late for that (which it might be đ€),
> should
> > we
> > > at least avoid introducing new "DAG" in the following PRs?
> > >
> > > Best,
> > > Wei
> > >
> > > On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > > https://github.com/apache/airflow/pull/43325
> > > > Let me know if that's the direction we want to go.
> > > >
> > > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > > <rb...@amazon.com.invalid> wrote:
> > > >
> > > > > I think Brent & Daniel summarized it best, "dag" is synonymous with
> > > > > workflows in Airflow through the way we talk and explain what
> Airflow
> > > is
> > > > > all about. Although folks would ask, I donât ever use the
> > mathematical
> > > > > definition of DAG.
> > > > > It will be challenging and possibly confusing for many users making
> > > such a
> > > > > change - I would rather direct that energy to appending the Oxford
> > > > > definition of "dag" to include a reference to workflows in Airflow.
> > __
> > > > >
> > > > > -- Rajesh
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > > > > jarek@potiuk.com>> wrote:
> > > > >
> > > > >
> > > > > CAUTION: This email originated from outside of the organization. Do
> > not
> > > > > click links or open attachments unless you can confirm the sender
> and
> > > know
> > > > > the content is safe.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
> > > externe.
> > > > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous
> ne
> > > pouvez
> > > > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> > certain
> > > que
> > > > > le contenu ne présente aucun risque.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From Guido's post:
> > > > >
> > > > >
> > > > > "Around this time the renaming seems to have been renamed".
> > > > >
> > > > >
> > > > > Naming is hard.
> > > > >
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev510e@gmail.com
> > > <mailto:
> > > > > droiddev510e@gmail.com>> wrote:
> > > > >
> > > > >
> > > > > > Yes, completely agree with above comments, dag is an Airflow term
> > now
> > > > > > rather
> > > > > > than just a "directed acyclic graph". Believe it or not, I've
> > worked
> > > with
> > > > > > work
> > > > > > folks who've been using dags in Airflow for years now and are pro
> > > devs
> > > > > but
> > > > > > have trouble remembering the full form of a DAG! For them, dag =
> > > Airflow.
> > > > > >
> > > > > > workflow, pipeline, flow are surely better terms but will be a
> > major
> > > > > > behavior
> > > > > > change for the developer community, so we'll need a solid plan on
> > > how to
> > > > > > introduce it, when we do.
> > > > > >
> > > > > > While we discuss this, I'd like to share about the Great Renaming
> > in
> > > core
> > > > > > Python started by Guido back in 2009. We could probably get some
> > > > > learnings
> > > > > > from there? Who knows!
> > > > > >
> > > > > > Guido's blog (2009):
> > > > > >
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > > <
> > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > >
> > > > > > Follow-up discussion (2024):
> > > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082
> <
> > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > > > >
> > > > > > Cheers,
> > > > > > Omkar
> > > > > >
> > > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com
> > > <mailto:
> > > > > weilee.rx@gmail.com>> wrote:
> > > > > >
> > > > > > > I think we should probably just accept it as an airflow term.
> At
> > > least,
> > > > > > > thatâs how I understood it when I first used Airflow. I feel
> > > renaming
> > > > > it
> > > > > > at
> > > > > > > this stage would require considerable effort from maintainers
> and
> > > > > > existing
> > > > > > > users without providing equivalent benefits.
> > > > > > >
> > > > > > > Best,
> > > > > > > Wei
> > > > > > >
> > > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxilnaik@gmail.com
> > > > > <ma...@gmail.com>> wrote:
> > > > > > > >
> > > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this
> > > discussion
> > > > > > for
> > > > > > > > Airflow 4 :) -- but right now it will cause too much
> disruption
> > > > > > > >
> > > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > > > <constance@astronomer.io.inva <mailto:
> > > constance@astronomer.io.inva>lid>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> In my experience, when you ask those with Airflow experience
> > > what a
> > > > > > dag
> > > > > > > is,
> > > > > > > >> they'll start talking about workflow attributes - stuff like
> > > dags
> > > > > > being
> > > > > > > a
> > > > > > > >> series of steps or tasks with owners. The structure doesn't
> > > come up.
> > > > > > > >>
> > > > > > > >> Echo-ing others, at this point, my vote is to embrace the
> name
> > > and
> > > > > > > >> de-emphasize the mathematical structure aspect.
> > > > > > > >>
> > > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > > <vikram@astronomer.io.inva <mailto:vikram@astronomer.io.inva
> > >lid>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> It's an interesting discussion and I remember struggling
> with
> > > this
> > > > > > > when I
> > > > > > > >>> started working with Airflow.
> > > > > > > >>> But, I also agree with the viewpoint of it being an
> > established
> > > > > > concept
> > > > > > > >> now
> > > > > > > >>> regardless of the origin.
> > > > > > > >>>
> > > > > > > >>> I am personally leaning towards the perspective best
> > expressed
> > > by
> > > > > > > Daniel
> > > > > > > >>> Standish and Brent of using Dag as a word, rather than the
> > > computer
> > > > > > > >> science
> > > > > > > >>> concept.
> > > > > > > >>>
> > > > > > > >>> Best regards,
> > > > > > > >>> Vikram
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > > > >> <onikolas@amazon.com.inva <mailto:onikolas@amazon.com.inva
> > >lid
> > > > > > > >>>>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> I agree with the general sentiment of: You're right Ryan,
> > DAG
> > > > > isn't
> > > > > > > >> great
> > > > > > > >>>> and I'd rather workflow, but changing it will cause much
> > more
> > > > > > wreckage
> > > > > > > >>> than
> > > > > > > >>>> it solves.
> > > > > > > >>>>
> > > > > > > >>>> Also agree with the idea to just move away from defining
> > DAG.
> > > I
> > > > > > think
> > > > > > > >>>> we've been naturally doing that as a community for a while
> > now
> > > > > > anyway,
> > > > > > > >> so
> > > > > > > >>>> that feels like a natural step.
> > > > > > > >>>>
> > > > > > > >>>> Cheers,
> > > > > > > >>>> Niko
> > > > > > > >>>>
> > > > > > > >>>> ________________________________
> > > > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > > ash@apache.org>>
> > > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > > >>>> To: dev@airflow.apache.org <mailto:dev@airflow.apache.org
> >
> > > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG"
> > > for end
> > > > > > > >> users
> > > > > > > >>>>
> > > > > > > >>>> CAUTION: This email originated from outside of the
> > > organization.
> > > > > Do
> > > > > > > not
> > > > > > > >>>> click links or open attachments unless you can confirm the
> > > sender
> > > > > > and
> > > > > > > >>> know
> > > > > > > >>>> the content is safe.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > > expéditeur
> > > > > > > >> externe.
> > > > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe
> si
> > > vous
> > > > > ne
> > > > > > > >>> pouvez
> > > > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
> > pas
> > > > > > certain
> > > > > > > >>> que
> > > > > > > >>>> le contenu ne présente aucun risque.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Best argument in favour of keeping âdagsâ as a term â
> > getting
> > > to
> > > > > > > re-use
> > > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > > > >>>>
> > > > > > > >>>> In all seriousness: I donât mind either way, both sides
> have
> > > good
> > > > > > > >> reasons
> > > > > > > >>>> presented.
> > > > > > > >>>>
> > > > > > > >>>> -a
> > > > > > > >>>>
> > > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>> Yeah just say, when asked where the name comes from,
> "well,
> > > no
> > > > > one
> > > > > > > >>>> actually
> > > > > > > >>>>> knows but..." and then make something up.
> > > > > > > >>>>>
> > > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > > jarek@potiuk.com
> > > > > <ma...@potiuk.com>>
> > > > > > > >> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > > > > tongue-twister,
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > > jarek@potiuk.com
> > > > > <ma...@potiuk.com>>
> > > > > > > >>> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very
> > much
> > > want
> > > > > > to
> > > > > > > >> be
> > > > > > > >>>>>> able
> > > > > > > >>>>>>> to say just "dag" without explaining it further.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > > > > tongue-twister,
> > > > > > > >>> and
> > > > > > > >>>> I
> > > > > > > >>>>>>> almost stutter on trying to recall how to pronounce it
> > > > > properly.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> J.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > > > >>>>>> <brent@astronomer.io.inva <mailto:
> > brent@astronomer.io.inva
> > > >lid>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> I remember we explored renaming "DAG" when starting on
> > > AIP-38
> > > > > to
> > > > > > > >>>>>> modernize
> > > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > > descriptive of
> > > > > > what
> > > > > > > >>> one
> > > > > > > >>>>>> is
> > > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > > > > implementation
> > > > > > > >>>> detail.
> > > > > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG"
> > has
> > > > > become
> > > > > > > >>> "dag"
> > > > > > > >>>>>> , a
> > > > > > > >>>>>>>> word in its own right.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Examples for "dag" are abound in community discussion,
> > > Airflow
> > > > > > > >>> Summit
> > > > > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace
> > > "dag".
> > > > > A
> > > > > > > >> user
> > > > > > > >>>>>> just
> > > > > > > >>>>>>>> needs to learn one new word vs the technical concept
> > > behind
> > > > > that
> > > > > > > >>>> word. I
> > > > > > > >>>>>>>> think that is much less effort than refactoring so
> much
> > > code,
> > > > > > > >>>>>>>> documentation, blog posts, stack overflow questions,
> > etc.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain
> > for
> > > > > > > >>>> questionable
> > > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I
> agree
> > > the
> > > > > > > >>>>>>>> association
> > > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> And along those lines, I offer here some less
> invasive
> > > > > > > >> mitigations.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize
> > the
> > > > > math
> > > > > > > >> nerd
> > > > > > > >>>>>>>> origins
> > > > > > > >>>>>>>>> of the name. That is to say, in docs / website / etc,
> > > *never
> > > > > > > >>> define*
> > > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph.
> > > Always
> > > > > > > >> define
> > > > > > > >>>> it
> > > > > > > >>>>>>>> as a
> > > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a
> > > historical
> > > > > > > >>>> footnote,
> > > > > > > >>>>>>>> and
> > > > > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> We could also start using lowercase in the docs in
> > > general
> > > > > e.g.
> > > > > > > >>>>>> writing
> > > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc.
> > The
> > > > > > upper
> > > > > > > >>> case
> > > > > > > >>>>>>>> part
> > > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in
> > > airlfow is
> > > > > > just
> > > > > > > >>> an
> > > > > > > >>>>>>>>> airflow concept and the association with "DAGs" is
> not
> > > really
> > > > > > > >>>>>> unhelpful.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its
> own
> > > > > thing,
> > > > > > is
> > > > > > > >>>>>>>>> *not* strictly
> > > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows
> > > about
> > > > > > > >>> anyway),
> > > > > > > >>>>>> and
> > > > > > > >>>>>>>>> tell them what it is in simple terms that normal
> people
> > > > > > > >> understand.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > > >
> > > > > > > >>>>>> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be
> > > extremely
> > > > > > > >>>>>>>> difficult to
> > > > > > > >>>>>>>>>> get rid of it completely. Also I think it will make
> a
> > > lot of
> > > > > > > >>>>>> "google"
> > > > > > > >>>>>>>>>> searches and "stack overflow" searches not finding
> the
> > > right
> > > > > > > >>>>>> answers.
> > > > > > > >>>>>>>>> This
> > > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the
> > > community
> > > > > and
> > > > > > > >>> ideas
> > > > > > > >>>>>>>> that
> > > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples,
> > > problems
> > > > > and
> > > > > > > >>>>>>>> solutions
> > > > > > > >>>>>>>>>> you can so easily find (and we have to remember that
> > > all the
> > > > > > AI
> > > > > > > >>>>>>>> trained
> > > > > > > >>>>>>>>> on
> > > > > > > >>>>>>>>>> past data will be also rather poorly matching
> queries
> > of
> > > > > > people.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch.
> > > And if
> > > > > we
> > > > > > > >> do
> > > > > > > >>> -
> > > > > > > >>>>>> I
> > > > > > > >>>>>>>>>> would be for using workflow or pipeline instead of
> > > `dag` if
> > > > > > not
> > > > > > > >>> the
> > > > > > > >>>>>>>> above
> > > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it
> might
> > > cause
> > > > > > more
> > > > > > > >>>>>>>> problems
> > > > > > > >>>>>>>>>> than it solves.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> But I am not 100% against - if others will think
> it's
> > a
> > > good
> > > > > > > >>> idea, I
> > > > > > > >>>>>>>> am
> > > > > > > >>>>>>>>> ok
> > > > > > > >>>>>>>>>> with it.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> J,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > > > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to
> > > someone
> > > > > new
> > > > > > > >> to
> > > > > > > >>>>>>>>> workflow
> > > > > > > >>>>>>>>>>> orchestration. But it does also show the nature of
> > > being
> > > > > > > >> acyclic.
> > > > > > > >>>>>>>> Sure,
> > > > > > > >>>>>>>>>> as
> > > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it.
> > Still,
> > > in
> > > > > my
> > > > > > > >>>>>>>> opinion,
> > > > > > > >>>>>>>>>> there
> > > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in
> workflow
> > > > > > > >>>>>> orchestration.
> > > > > > > >>>>>>>>> Most
> > > > > > > >>>>>>>>>>> (*if
> > > > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered
> > > using an
> > > > > > > >>> acyclic
> > > > > > > >>>>>>>>> manner
> > > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I
> > would
> > > want
> > > > > > the
> > > > > > > >>>>>>>>> "acyclic"
> > > > > > > >>>>>>>>>>> word to stick.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Regards,
> > > > > > > >>>>>>>>>>> Avi
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > > > > >>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow
> Fan;
> > > so
> > > > > many
> > > > > > > >> new
> > > > > > > >>>>>>>>> ideas.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced
> by
> > > the
> > > > > > more
> > > > > > > >>>>>>>> general
> > > > > > > >>>>>>>>>>> Term
> > > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri)
> > Net
> > > > > (AFN)
> > > > > > > >>>>>> (maybe
> > > > > > > >>>>>>>>>>> without
> > > > > > > >>>>>>>>>>>> a direction);
> > > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph
> > > Database.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG)
> > > might be
> > > > > > > >>>>>>>> assigned
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>> an
> > > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a
> > library.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> A running Graph might have a different structure
> > than
> > > a
> > > > > > > >>>>>>>> configuration
> > > > > > > >>>>>>>>>>>> Graph.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Best Regards
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > > bernd.stroehle@gmail.com>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA
> > > <mailto:
> > > > > ikholopov@google.com.INVA>LID>
> > > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > > dev@airflow.apache.org
> > > > > >
> > > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term
> "DAG"
> > > for
> > > > > end
> > > > > > > >>>>>> users
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal,
> it
> > > is
> > > > > part
> > > > > > > >> of
> > > > > > > >>>>>>>>> Airflow
> > > > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that
> any
> > > > > attempt
> > > > > > > >> to
> > > > > > > >>>>>>>>> change
> > > > > > > >>>>>>>>>> it
> > > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The
> > > only
> > > > > > thing
> > > > > > > >>>>>>>> that is
> > > > > > > >>>>>>>>>>> worse
> > > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when
> there
> > > are
> > > > > two
> > > > > > > >> ways
> > > > > > > >>>>>>>> to
> > > > > > > >>>>>>>>>>> define
> > > > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials
> > will
> > > > > > suddenly
> > > > > > > >>>>>>>> become
> > > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as
> "DAG"s.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> We are already at risk of scaring the users away
> > with
> > > a
> > > > > > number
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>>>> breaking
> > > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking
> > > changes
> > > > > > for
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>> most
> > > > > > > >>>>>>>>>>>> basic things is not something that people are
> > looking
> > > for.
> > > > > > > >>>>>>>> Attempting
> > > > > > > >>>>>>>>>> to
> > > > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted
> as
> > an
> > > > > even
> > > > > > > >>>>>>>> stronger
> > > > > > > >>>>>>>>>>> signal
> > > > > > > >>>>>>>>>>>> of project immaturity.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term
> > in
> > > the
> > > > > > long
> > > > > > > >>>>>>>> run. I
> > > > > > > >>>>>>>>>>> even
> > > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the
> > DAG
> > > > > > > >> definition
> > > > > > > >>>>>>>>>>> interface.
> > > > > > > >>>>>>>>>>>> We better put our time and efforts in other places
> > in
> > > > > > Airflow,
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>> which
> > > > > > > >>>>>>>>>>>> there are plenty.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Kind regards,
> > > > > > > >>>>>>>>>>>> Igor
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:
> > bas@astronomer.io.inva
> > > >lid
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have
> already
> > > faded
> > > > > > over
> > > > > > > >>>>>>>> time,
> > > > > > > >>>>>>>>>>>>> for example there are now several ways to create
> > > cyclic
> > > > > > > >>>>>> graphs,
> > > > > > > >>>>>>>>> e.g.
> > > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these
> > > > > properties
> > > > > > > >>>>>>>>> vanishing
> > > > > > > >>>>>>>>>>>>> even more in the future, so from that
> perspective I
> > > > > support
> > > > > > > >>>>>>>>> changing
> > > > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming:
> > > > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I
> suggest
> > > > > > defining
> > > > > > > >> a
> > > > > > > >>>>>>>>> clear
> > > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the
> > > deprecation in
> > > > > > > >>>>>>>> Airflow
> > > > > > > >>>>>>>>> 3,
> > > > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Bas
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> > neil4real@gmail.com
> > > > > <ma...@gmail.com>>
> > > > > > wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG,
> > especially
> > > when
> > > > > > > >>>>>> most
> > > > > > > >>>>>>>>> other
> > > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing
> > > about it
> > > > > at
> > > > > > > >>>>>>>> all,
> > > > > > > >>>>>>>>>>>>>> changing the term though would be fairly
> confusing
> > > to
> > > > > most
> > > > > > > >>>>>>>> users
> > > > > > > >>>>>>>>>> who
> > > > > > > >>>>>>>>>>>>>> have been
> > > > > > > >>>>>>>>>>>>> using
> > > > > > > >>>>>>>>>>>>>> the term for years.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung
> > > > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > > tp@astronomer.io.inva
> > > > > >lid
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term
> > DAG.
> > > The
> > > > > > only
> > > > > > > >>>>>>>>>> problem
> > > > > > > >>>>>>>>>>>>> (aside
> > > > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding
> myselfâto
> > > stop
> > > > > > using
> > > > > > > >>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> term)
> > > > > > > >>>>>>>>>>>>> is to
> > > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned
> > > âworkflowâ is
> > > > > > not
> > > > > > > >>>>>>>> very
> > > > > > > >>>>>>>>>>>>> accurate
> > > > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was
> proposed,
> > > but
> > > > > > itâs a
> > > > > > > >>>>>>>> bit
> > > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and
> kept
> > > > > finding
> > > > > > > >>>>>>>> myself
> > > > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to
> > clarify
> > > âDAG
> > > > > > > >>>>>>>>>> definitionâ
> > > > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> TP
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > > > j_scheffler@gmx.de.INVA>LID>
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same
> > > > > > observation,
> > > > > > > >>>>>>>> had a
> > > > > > > >>>>>>>>>>>>>>>> short
> > > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an
> > > > > introduction
> > > > > > > >>>>>> if
> > > > > > > >>>>>>>>>>>>>>> someone
> > > > > > > >>>>>>>>>>>>> joins
> > > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes
> sense.
> > > > > > > >>>>>> Especially
> > > > > > > >>>>>>>>> when
> > > > > > > >>>>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a
> reasonable
> > > step.
> > > > > > > >>>>>>>> Concepts
> > > > > > > >>>>>>>>>>>>>>> still
> > > > > > > >>>>>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding below
> the
> > > desk
> > > > > > with
> > > > > > > >>>>>>>> you
> > > > > > > >>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>> +1
> > > > > > > >>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep
> > details
> > > of
> > > > > > > >>>>>> python
> > > > > > > >>>>>>>>> names
> > > > > > > >>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAG⊠but
> > user
> > > > > > facing
> > > > > > > >>>>>> it
> > > > > > > >>>>>>>>> is a
> > > > > > > >>>>>>>>>>>>> workflow.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Jens
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > > ryan.hatter@astronomer.io
> > > > > >
> > > > > > > >>>>>>>>>>>>> .invalid>
> > > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at
> least
> > > for
> > > > > > now.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to
> > Airflow
> > > > > > users.
> > > > > > > >>>>>>>>> Indeed,
> > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>> has
> > > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians
> > and
> > > > > > > >>>>>> software
> > > > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > > > >>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter.
> > For
> > > > > > someone
> > > > > > > >>>>>>>> new
> > > > > > > >>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration,
> one
> > > of
> > > > > the
> > > > > > > >>>>>>>> first
> > > > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > > > >>>>>>>>>>>>>>> will
> > > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The
> > > answer
> > > > > is
> > > > > > > >>>>>>>> almost
> > > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You
> > don't
> > > > > need
> > > > > > to
> > > > > > > >>>>>>>>> worry
> > > > > > > >>>>>>>>>>>>>>>>> about what
> > > > > > > >>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow."
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a
> > > couple
> > > > > > > >>>>>>>> important
> > > > > > > >>>>>>>>>>>> reasons:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned
> above,
> > > "DAG"
> > > > > > is
> > > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We
> > want
> > > > > > Airflow
> > > > > > > >>>>>>>> to
> > > > > > > >>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > > > >>>>>>>>>>>>>>> using
> > > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat
> > > creates
> > > > > > an
> > > > > > > >>>>>>>>> initial
> > > > > > > >>>>>>>>>>>>>>> barrier to
> > > > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow
> Concepts*:
> > > The
> > > > > DAG
> > > > > > > >>>>>> is
> > > > > > > >>>>>>>>> just
> > > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The
> > > workflow
> > > > > > > >>>>>> includes
> > > > > > > >>>>>>>>> its
> > > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other
> > > > > parameters,
> > > > > > > >>>>>> and
> > > > > > > >>>>>>>>> other
> > > > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > > > >>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow
> > homepage
> > > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > > > https://airflow.apache.org/>>.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by the
> > > > > community
> > > > > > to
> > > > > > > >>>>>>>>>>>>>>> programmatically
> > > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?"
> docs
> > > page
> > > > > > > >>>>>>>>>>>>>>>>> <
> > > > > > > >>>>>>>>>>
> > > > > > > >>
> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html>
> > > > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > > > >>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is
> > without
> > > > > using
> > > > > > > >>>>>>>> "DAG."
> > > > > > > >>>>>>>>>>>>>>>>> It's
> > > > > > > >>>>>>>>>>>>>>> only in
> > > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is
> > > > > introduced
> > > > > > > >>>>>> out
> > > > > > > >>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > > > >>>>>>>>>>>>>>> as a
> > > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection
> of
> > > tasks
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these
> > > > > > > >>>>>> introductions
> > > > > > > >>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate
> > > DAGs;
> > > > > it
> > > > > > > >>>>>>>>>>> orchestrates
> > > > > > > >>>>>>>>>>>> workflows*.
> > > > > > > >>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant
> > to
> > > > > almost
> > > > > > > >>>>>>>> every
> > > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for
> the
> > > term
> > > > > > "DAG"
> > > > > > > >>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with
> > > > > > "workflow".
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>
> > > ------------------------------------------------------------------
> > > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > > > dev-help@airflow.apache.org>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>
> > > -------------------------------------------------------------------
> > > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > > >>>>>> dev-help@airflow.apache.org <mailto:
> > > dev-help@airflow.apache.org
> > > > > >
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > <mailto:
> > > > > dev-unsubscribe@airflow.apache.org>
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > <mailto:
> > > > > dev-help@airflow.apache.org>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Omkar P <dr...@gmail.com>.
+1 for ruff rules :)
Also would be nice to introduce 'Dag' to replace 'DAG' in the Airflow
docs, in line with the new UI changes and to make the renaming
consistent across user-facing pages.
Regards,
Omkar
On Mon, Feb 17, 2025 at 8:12âŻAM Jarek Potiuk <ja...@potiuk.com> wrote:
> +1 :) . Maybe we could add a ruff rule for that :)
>
> On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org> wrote:
>
> > It seems that our current conclusion is to use "dag" or "Dag" instead of
> > "DAG" whenever possible. Should we replace all "DAG" in the codebase with
> > "dag" or "Dag"? If it's too late for that (which it might be đ€), should
> we
> > at least avoid introducing new "DAG" in the following PRs?
> >
> > Best,
> > Wei
> >
> > On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > > https://github.com/apache/airflow/pull/43325
> > > Let me know if that's the direction we want to go.
> > >
> > > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > > <rb...@amazon.com.invalid> wrote:
> > >
> > > > I think Brent & Daniel summarized it best, "dag" is synonymous with
> > > > workflows in Airflow through the way we talk and explain what Airflow
> > is
> > > > all about. Although folks would ask, I donât ever use the
> mathematical
> > > > definition of DAG.
> > > > It will be challenging and possibly confusing for many users making
> > such a
> > > > change - I would rather direct that energy to appending the Oxford
> > > > definition of "dag" to include a reference to workflows in Airflow.
> __
> > > >
> > > > -- Rajesh
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > > > jarek@potiuk.com>> wrote:
> > > >
> > > >
> > > > CAUTION: This email originated from outside of the organization. Do
> not
> > > > click links or open attachments unless you can confirm the sender and
> > know
> > > > the content is safe.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
> > externe.
> > > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne
> > pouvez
> > > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> certain
> > que
> > > > le contenu ne présente aucun risque.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From Guido's post:
> > > >
> > > >
> > > > "Around this time the renaming seems to have been renamed".
> > > >
> > > >
> > > > Naming is hard.
> > > >
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev510e@gmail.com
> > <mailto:
> > > > droiddev510e@gmail.com>> wrote:
> > > >
> > > >
> > > > > Yes, completely agree with above comments, dag is an Airflow term
> now
> > > > > rather
> > > > > than just a "directed acyclic graph". Believe it or not, I've
> worked
> > with
> > > > > work
> > > > > folks who've been using dags in Airflow for years now and are pro
> > devs
> > > > but
> > > > > have trouble remembering the full form of a DAG! For them, dag =
> > Airflow.
> > > > >
> > > > > workflow, pipeline, flow are surely better terms but will be a
> major
> > > > > behavior
> > > > > change for the developer community, so we'll need a solid plan on
> > how to
> > > > > introduce it, when we do.
> > > > >
> > > > > While we discuss this, I'd like to share about the Great Renaming
> in
> > core
> > > > > Python started by Guido back in 2009. We could probably get some
> > > > learnings
> > > > > from there? Who knows!
> > > > >
> > > > > Guido's blog (2009):
> > > > >
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > > <
> > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> >
> > > > > Follow-up discussion (2024):
> > > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 <
> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > > >
> > > > > Cheers,
> > > > > Omkar
> > > > >
> > > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com
> > <mailto:
> > > > weilee.rx@gmail.com>> wrote:
> > > > >
> > > > > > I think we should probably just accept it as an airflow term. At
> > least,
> > > > > > thatâs how I understood it when I first used Airflow. I feel
> > renaming
> > > > it
> > > > > at
> > > > > > this stage would require considerable effort from maintainers and
> > > > > existing
> > > > > > users without providing equivalent benefits.
> > > > > >
> > > > > > Best,
> > > > > > Wei
> > > > > >
> > > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxilnaik@gmail.com
> > > > <ma...@gmail.com>> wrote:
> > > > > > >
> > > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this
> > discussion
> > > > > for
> > > > > > > Airflow 4 :) -- but right now it will cause too much disruption
> > > > > > >
> > > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > > <constance@astronomer.io.inva <mailto:
> > constance@astronomer.io.inva>lid>
> > > > wrote:
> > > > > > >
> > > > > > >> In my experience, when you ask those with Airflow experience
> > what a
> > > > > dag
> > > > > > is,
> > > > > > >> they'll start talking about workflow attributes - stuff like
> > dags
> > > > > being
> > > > > > a
> > > > > > >> series of steps or tasks with owners. The structure doesn't
> > come up.
> > > > > > >>
> > > > > > >> Echo-ing others, at this point, my vote is to embrace the name
> > and
> > > > > > >> de-emphasize the mathematical structure aspect.
> > > > > > >>
> > > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > > <vikram@astronomer.io.inva <mailto:vikram@astronomer.io.inva
> >lid>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> It's an interesting discussion and I remember struggling with
> > this
> > > > > > when I
> > > > > > >>> started working with Airflow.
> > > > > > >>> But, I also agree with the viewpoint of it being an
> established
> > > > > concept
> > > > > > >> now
> > > > > > >>> regardless of the origin.
> > > > > > >>>
> > > > > > >>> I am personally leaning towards the perspective best
> expressed
> > by
> > > > > > Daniel
> > > > > > >>> Standish and Brent of using Dag as a word, rather than the
> > computer
> > > > > > >> science
> > > > > > >>> concept.
> > > > > > >>>
> > > > > > >>> Best regards,
> > > > > > >>> Vikram
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > > >> <onikolas@amazon.com.inva <mailto:onikolas@amazon.com.inva
> >lid
> > > > > > >>>>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> I agree with the general sentiment of: You're right Ryan,
> DAG
> > > > isn't
> > > > > > >> great
> > > > > > >>>> and I'd rather workflow, but changing it will cause much
> more
> > > > > wreckage
> > > > > > >>> than
> > > > > > >>>> it solves.
> > > > > > >>>>
> > > > > > >>>> Also agree with the idea to just move away from defining
> DAG.
> > I
> > > > > think
> > > > > > >>>> we've been naturally doing that as a community for a while
> now
> > > > > anyway,
> > > > > > >> so
> > > > > > >>>> that feels like a natural step.
> > > > > > >>>>
> > > > > > >>>> Cheers,
> > > > > > >>>> Niko
> > > > > > >>>>
> > > > > > >>>> ________________________________
> > > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> > ash@apache.org>>
> > > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > > >>>> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG"
> > for end
> > > > > > >> users
> > > > > > >>>>
> > > > > > >>>> CAUTION: This email originated from outside of the
> > organization.
> > > > Do
> > > > > > not
> > > > > > >>>> click links or open attachments unless you can confirm the
> > sender
> > > > > and
> > > > > > >>> know
> > > > > > >>>> the content is safe.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> > expéditeur
> > > > > > >> externe.
> > > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si
> > vous
> > > > ne
> > > > > > >>> pouvez
> > > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes
> pas
> > > > > certain
> > > > > > >>> que
> > > > > > >>>> le contenu ne présente aucun risque.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Best argument in favour of keeping âdagsâ as a term â
> getting
> > to
> > > > > > re-use
> > > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > > >>>>
> > > > > > >>>> In all seriousness: I donât mind either way, both sides have
> > good
> > > > > > >> reasons
> > > > > > >>>> presented.
> > > > > > >>>>
> > > > > > >>>> -a
> > > > > > >>>>
> > > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Yeah just say, when asked where the name comes from, "well,
> > no
> > > > one
> > > > > > >>>> actually
> > > > > > >>>>> knows but..." and then make something up.
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> > jarek@potiuk.com
> > > > <ma...@potiuk.com>>
> > > > > > >> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > > > tongue-twister,
> > > > > > >>>>>>
> > > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> > jarek@potiuk.com
> > > > <ma...@potiuk.com>>
> > > > > > >>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very
> much
> > want
> > > > > to
> > > > > > >> be
> > > > > > >>>>>> able
> > > > > > >>>>>>> to say just "dag" without explaining it further.
> > > > > > >>>>>>>
> > > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > > > tongue-twister,
> > > > > > >>> and
> > > > > > >>>> I
> > > > > > >>>>>>> almost stutter on trying to recall how to pronounce it
> > > > properly.
> > > > > > >>>>>>>
> > > > > > >>>>>>> J.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > > >>>>>> <brent@astronomer.io.inva <mailto:
> brent@astronomer.io.inva
> > >lid>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> I remember we explored renaming "DAG" when starting on
> > AIP-38
> > > > to
> > > > > > >>>>>> modernize
> > > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> > descriptive of
> > > > > what
> > > > > > >>> one
> > > > > > >>>>>> is
> > > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > > > implementation
> > > > > > >>>> detail.
> > > > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG"
> has
> > > > become
> > > > > > >>> "dag"
> > > > > > >>>>>> , a
> > > > > > >>>>>>>> word in its own right.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Examples for "dag" are abound in community discussion,
> > Airflow
> > > > > > >>> Summit
> > > > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace
> > "dag".
> > > > A
> > > > > > >> user
> > > > > > >>>>>> just
> > > > > > >>>>>>>> needs to learn one new word vs the technical concept
> > behind
> > > > that
> > > > > > >>>> word. I
> > > > > > >>>>>>>> think that is much less effort than refactoring so much
> > code,
> > > > > > >>>>>>>> documentation, blog posts, stack overflow questions,
> etc.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain
> for
> > > > > > >>>> questionable
> > > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I agree
> > the
> > > > > > >>>>>>>> association
> > > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> And along those lines, I offer here some less invasive
> > > > > > >> mitigations.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize
> the
> > > > math
> > > > > > >> nerd
> > > > > > >>>>>>>> origins
> > > > > > >>>>>>>>> of the name. That is to say, in docs / website / etc,
> > *never
> > > > > > >>> define*
> > > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph.
> > Always
> > > > > > >> define
> > > > > > >>>> it
> > > > > > >>>>>>>> as a
> > > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a
> > historical
> > > > > > >>>> footnote,
> > > > > > >>>>>>>> and
> > > > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> We could also start using lowercase in the docs in
> > general
> > > > e.g.
> > > > > > >>>>>> writing
> > > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc.
> The
> > > > > upper
> > > > > > >>> case
> > > > > > >>>>>>>> part
> > > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in
> > airlfow is
> > > > > just
> > > > > > >>> an
> > > > > > >>>>>>>>> airflow concept and the association with "DAGs" is not
> > really
> > > > > > >>>>>> unhelpful.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its own
> > > > thing,
> > > > > is
> > > > > > >>>>>>>>> *not* strictly
> > > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows
> > about
> > > > > > >>> anyway),
> > > > > > >>>>>> and
> > > > > > >>>>>>>>> tell them what it is in simple terms that normal people
> > > > > > >> understand.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > >
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be
> > extremely
> > > > > > >>>>>>>> difficult to
> > > > > > >>>>>>>>>> get rid of it completely. Also I think it will make a
> > lot of
> > > > > > >>>>>> "google"
> > > > > > >>>>>>>>>> searches and "stack overflow" searches not finding the
> > right
> > > > > > >>>>>> answers.
> > > > > > >>>>>>>>> This
> > > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the
> > community
> > > > and
> > > > > > >>> ideas
> > > > > > >>>>>>>> that
> > > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples,
> > problems
> > > > and
> > > > > > >>>>>>>> solutions
> > > > > > >>>>>>>>>> you can so easily find (and we have to remember that
> > all the
> > > > > AI
> > > > > > >>>>>>>> trained
> > > > > > >>>>>>>>> on
> > > > > > >>>>>>>>>> past data will be also rather poorly matching queries
> of
> > > > > people.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch.
> > And if
> > > > we
> > > > > > >> do
> > > > > > >>> -
> > > > > > >>>>>> I
> > > > > > >>>>>>>>>> would be for using workflow or pipeline instead of
> > `dag` if
> > > > > not
> > > > > > >>> the
> > > > > > >>>>>>>> above
> > > > > > >>>>>>>>>> reason, but I think I am here with Igor that it might
> > cause
> > > > > more
> > > > > > >>>>>>>> problems
> > > > > > >>>>>>>>>> than it solves.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> But I am not 100% against - if others will think it's
> a
> > good
> > > > > > >>> idea, I
> > > > > > >>>>>>>> am
> > > > > > >>>>>>>>> ok
> > > > > > >>>>>>>>>> with it.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> J,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to
> > someone
> > > > new
> > > > > > >> to
> > > > > > >>>>>>>>> workflow
> > > > > > >>>>>>>>>>> orchestration. But it does also show the nature of
> > being
> > > > > > >> acyclic.
> > > > > > >>>>>>>> Sure,
> > > > > > >>>>>>>>>> as
> > > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it.
> Still,
> > in
> > > > my
> > > > > > >>>>>>>> opinion,
> > > > > > >>>>>>>>>> there
> > > > > > >>>>>>>>>>> is generally no need for cyclic behavior in workflow
> > > > > > >>>>>> orchestration.
> > > > > > >>>>>>>>> Most
> > > > > > >>>>>>>>>>> (*if
> > > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered
> > using an
> > > > > > >>> acyclic
> > > > > > >>>>>>>>> manner
> > > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I
> would
> > want
> > > > > the
> > > > > > >>>>>>>>> "acyclic"
> > > > > > >>>>>>>>>>> word to stick.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Regards,
> > > > > > >>>>>>>>>>> Avi
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow Fan;
> > so
> > > > many
> > > > > > >> new
> > > > > > >>>>>>>>> ideas.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced by
> > the
> > > > > more
> > > > > > >>>>>>>> general
> > > > > > >>>>>>>>>>> Term
> > > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri)
> Net
> > > > (AFN)
> > > > > > >>>>>> (maybe
> > > > > > >>>>>>>>>>> without
> > > > > > >>>>>>>>>>>> a direction);
> > > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph
> > Database.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG)
> > might be
> > > > > > >>>>>>>> assigned
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>> an
> > > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a
> library.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> A running Graph might have a different structure
> than
> > a
> > > > > > >>>>>>>> configuration
> > > > > > >>>>>>>>>>>> Graph.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best Regards
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > > bernd.stroehle@gmail.com>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA
> > <mailto:
> > > > ikholopov@google.com.INVA>LID>
> > > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> > dev@airflow.apache.org
> > > > >
> > > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term "DAG"
> > for
> > > > end
> > > > > > >>>>>> users
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, it
> > is
> > > > part
> > > > > > >> of
> > > > > > >>>>>>>>> Airflow
> > > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that any
> > > > attempt
> > > > > > >> to
> > > > > > >>>>>>>>> change
> > > > > > >>>>>>>>>> it
> > > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The
> > only
> > > > > thing
> > > > > > >>>>>>>> that is
> > > > > > >>>>>>>>>>> worse
> > > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when there
> > are
> > > > two
> > > > > > >> ways
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>>>>> define
> > > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials
> will
> > > > > suddenly
> > > > > > >>>>>>>> become
> > > > > > >>>>>>>>>>>> confusing as they all refer to workflows as "DAG"s.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> We are already at risk of scaring the users away
> with
> > a
> > > > > number
> > > > > > >>>>>> of
> > > > > > >>>>>>>>>>> breaking
> > > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking
> > changes
> > > > > for
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> most
> > > > > > >>>>>>>>>>>> basic things is not something that people are
> looking
> > for.
> > > > > > >>>>>>>> Attempting
> > > > > > >>>>>>>>>> to
> > > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted as
> an
> > > > even
> > > > > > >>>>>>>> stronger
> > > > > > >>>>>>>>>>> signal
> > > > > > >>>>>>>>>>>> of project immaturity.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term
> in
> > the
> > > > > long
> > > > > > >>>>>>>> run. I
> > > > > > >>>>>>>>>>> even
> > > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the
> DAG
> > > > > > >> definition
> > > > > > >>>>>>>>>>> interface.
> > > > > > >>>>>>>>>>>> We better put our time and efforts in other places
> in
> > > > > Airflow,
> > > > > > >>>>>> of
> > > > > > >>>>>>>>> which
> > > > > > >>>>>>>>>>>> there are plenty.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Kind regards,
> > > > > > >>>>>>>>>>>> Igor
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:
> bas@astronomer.io.inva
> > >lid
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have already
> > faded
> > > > > over
> > > > > > >>>>>>>> time,
> > > > > > >>>>>>>>>>>>> for example there are now several ways to create
> > cyclic
> > > > > > >>>>>> graphs,
> > > > > > >>>>>>>>> e.g.
> > > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these
> > > > properties
> > > > > > >>>>>>>>> vanishing
> > > > > > >>>>>>>>>>>>> even more in the future, so from that perspective I
> > > > support
> > > > > > >>>>>>>>> changing
> > > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming:
> > > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I suggest
> > > > > defining
> > > > > > >> a
> > > > > > >>>>>>>>> clear
> > > > > > >>>>>>>>>>>>> path forward. Would we first introduce the
> > deprecation in
> > > > > > >>>>>>>> Airflow
> > > > > > >>>>>>>>> 3,
> > > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Bas
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <
> neil4real@gmail.com
> > > > <ma...@gmail.com>>
> > > > > wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG,
> especially
> > when
> > > > > > >>>>>> most
> > > > > > >>>>>>>>> other
> > > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing
> > about it
> > > > at
> > > > > > >>>>>>>> all,
> > > > > > >>>>>>>>>>>>>> changing the term though would be fairly confusing
> > to
> > > > most
> > > > > > >>>>>>>> users
> > > > > > >>>>>>>>>> who
> > > > > > >>>>>>>>>>>>>> have been
> > > > > > >>>>>>>>>>>>> using
> > > > > > >>>>>>>>>>>>>> the term for years.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung
> > > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> > tp@astronomer.io.inva
> > > > >lid
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term
> DAG.
> > The
> > > > > only
> > > > > > >>>>>>>>>> problem
> > > > > > >>>>>>>>>>>>> (aside
> > > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding myselfâto
> > stop
> > > > > using
> > > > > > >>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> term)
> > > > > > >>>>>>>>>>>>> is to
> > > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned
> > âworkflowâ is
> > > > > not
> > > > > > >>>>>>>> very
> > > > > > >>>>>>>>>>>>> accurate
> > > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was proposed,
> > but
> > > > > itâs a
> > > > > > >>>>>>>> bit
> > > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and kept
> > > > finding
> > > > > > >>>>>>>> myself
> > > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to
> clarify
> > âDAG
> > > > > > >>>>>>>>>> definitionâ
> > > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> TP
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > > j_scheffler@gmx.de.INVA>LID>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same
> > > > > observation,
> > > > > > >>>>>>>> had a
> > > > > > >>>>>>>>>>>>>>>> short
> > > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an
> > > > introduction
> > > > > > >>>>>> if
> > > > > > >>>>>>>>>>>>>>> someone
> > > > > > >>>>>>>>>>>>> joins
> > > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes sense.
> > > > > > >>>>>> Especially
> > > > > > >>>>>>>>> when
> > > > > > >>>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a reasonable
> > step.
> > > > > > >>>>>>>> Concepts
> > > > > > >>>>>>>>>>>>>>> still
> > > > > > >>>>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding below the
> > desk
> > > > > with
> > > > > > >>>>>>>> you
> > > > > > >>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>> +1
> > > > > > >>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep
> details
> > of
> > > > > > >>>>>> python
> > > > > > >>>>>>>>> names
> > > > > > >>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>> same because the execution is still a DAG⊠but
> user
> > > > > facing
> > > > > > >>>>>> it
> > > > > > >>>>>>>>> is a
> > > > > > >>>>>>>>>>>>> workflow.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Jens
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> > ryan.hatter@astronomer.io
> > > > >
> > > > > > >>>>>>>>>>>>> .invalid>
> > > > > > >>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at least
> > for
> > > > > now.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to
> Airflow
> > > > > users.
> > > > > > >>>>>>>>> Indeed,
> > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>> has
> > > > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians
> and
> > > > > > >>>>>> software
> > > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > > >>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter.
> For
> > > > > someone
> > > > > > >>>>>>>> new
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, one
> > of
> > > > the
> > > > > > >>>>>>>> first
> > > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > > >>>>>>>>>>>>>>> will
> > > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The
> > answer
> > > > is
> > > > > > >>>>>>>> almost
> > > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You
> don't
> > > > need
> > > > > to
> > > > > > >>>>>>>>> worry
> > > > > > >>>>>>>>>>>>>>>>> about what
> > > > > > >>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow."
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a
> > couple
> > > > > > >>>>>>>> important
> > > > > > >>>>>>>>>>>> reasons:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned above,
> > "DAG"
> > > > > is
> > > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We
> want
> > > > > Airflow
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > > >>>>>>>>>>>>>>> using
> > > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat
> > creates
> > > > > an
> > > > > > >>>>>>>>> initial
> > > > > > >>>>>>>>>>>>>>> barrier to
> > > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow Concepts*:
> > The
> > > > DAG
> > > > > > >>>>>> is
> > > > > > >>>>>>>>> just
> > > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The
> > workflow
> > > > > > >>>>>> includes
> > > > > > >>>>>>>>> its
> > > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other
> > > > parameters,
> > > > > > >>>>>> and
> > > > > > >>>>>>>>> other
> > > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > > >>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow
> homepage
> > > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > > https://airflow.apache.org/>>.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by the
> > > > community
> > > > > to
> > > > > > >>>>>>>>>>>>>>> programmatically
> > > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" docs
> > page
> > > > > > >>>>>>>>>>>>>>>>> <
> > > > > > >>>>>>>>>>
> > > > > > >>
> > https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > > https://airflow.apache.org/docs/apache-airflow/stable/index.html>
> > > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > > >>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is
> without
> > > > using
> > > > > > >>>>>>>> "DAG."
> > > > > > >>>>>>>>>>>>>>>>> It's
> > > > > > >>>>>>>>>>>>>>> only in
> > > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is
> > > > introduced
> > > > > > >>>>>> out
> > > > > > >>>>>>>> of
> > > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > > >>>>>>>>>>>>>>> as a
> > > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection of
> > tasks
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these
> > > > > > >>>>>> introductions
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate
> > DAGs;
> > > > it
> > > > > > >>>>>>>>>>> orchestrates
> > > > > > >>>>>>>>>>>> workflows*.
> > > > > > >>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant
> to
> > > > almost
> > > > > > >>>>>>>> every
> > > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for the
> > term
> > > > > "DAG"
> > > > > > >>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with
> > > > > "workflow".
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > > >>>>>>>>>>>>>>>>> ...
> > > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > > >>>>>>>>>>>>>>>>> )
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>
> > ------------------------------------------------------------------
> > > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > dev-unsubscribe@airflow.apache.org>
> > > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > > dev-help@airflow.apache.org>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>
> > -------------------------------------------------------------------
> > > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > > dev-unsubscribe@airflow.apache.org>
> > > > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > > >>>>>> dev-help@airflow.apache.org <mailto:
> > dev-help@airflow.apache.org
> > > > >
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>
> > > > ---------------------------------------------------------------------
> > > > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > > > dev-unsubscribe@airflow.apache.org <mailto:
> > > > dev-unsubscribe@airflow.apache.org>
> > > > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > <mailto:
> > > > dev-unsubscribe@airflow.apache.org>
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > <mailto:
> > > > dev-help@airflow.apache.org>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: Airflow should deprecate the term "DAG" for end users
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 :) . Maybe we could add a ruff rule for that :)
On Mon, Feb 17, 2025 at 8:27âŻAM Wei Lee <we...@apache.org> wrote:
> It seems that our current conclusion is to use "dag" or "Dag" instead of
> "DAG" whenever possible. Should we replace all "DAG" in the codebase with
> "dag" or "Dag"? If it's too late for that (which it might be đ€), should we
> at least avoid introducing new "DAG" in the following PRs?
>
> Best,
> Wei
>
> On 2024/10/23 17:16:55 Brent Bovenzi wrote:
> > Here's a PR to use "dag" as a word in the new 3.0 UI:
> > https://github.com/apache/airflow/pull/43325
> > Let me know if that's the direction we want to go.
> >
> > On Wed, Oct 23, 2024 at 11:06âŻAM Bishundeo, Rajeshwar
> > <rb...@amazon.com.invalid> wrote:
> >
> > > I think Brent & Daniel summarized it best, "dag" is synonymous with
> > > workflows in Airflow through the way we talk and explain what Airflow
> is
> > > all about. Although folks would ask, I donât ever use the mathematical
> > > definition of DAG.
> > > It will be challenging and possibly confusing for many users making
> such a
> > > change - I would rather direct that energy to appending the Oxford
> > > definition of "dag" to include a reference to workflows in Airflow. __
> > >
> > > -- Rajesh
> > >
> > >
> > >
> > >
> > >
> > >
> > > ï»żOn 2024-10-23, 3:37 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> > > jarek@potiuk.com>> wrote:
> > >
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > > click links or open attachments unless you can confirm the sender and
> know
> > > the content is safe.
> > >
> > >
> > >
> > >
> > >
> > >
> > > AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun expĂ©diteur
> externe.
> > > Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si vous ne
> pouvez
> > > pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas certain
> que
> > > le contenu ne présente aucun risque.
> > >
> > >
> > >
> > >
> > >
> > >
> > > From Guido's post:
> > >
> > >
> > > "Around this time the renaming seems to have been renamed".
> > >
> > >
> > > Naming is hard.
> > >
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Wed, Oct 23, 2024 at 7:11 AM Omkar P <droiddev510e@gmail.com
> <mailto:
> > > droiddev510e@gmail.com>> wrote:
> > >
> > >
> > > > Yes, completely agree with above comments, dag is an Airflow term now
> > > > rather
> > > > than just a "directed acyclic graph". Believe it or not, I've worked
> with
> > > > work
> > > > folks who've been using dags in Airflow for years now and are pro
> devs
> > > but
> > > > have trouble remembering the full form of a DAG! For them, dag =
> Airflow.
> > > >
> > > > workflow, pipeline, flow are surely better terms but will be a major
> > > > behavior
> > > > change for the developer community, so we'll need a solid plan on
> how to
> > > > introduce it, when we do.
> > > >
> > > > While we discuss this, I'd like to share about the Great Renaming in
> core
> > > > Python started by Guido back in 2009. We could probably get some
> > > learnings
> > > > from there? Who knows!
> > > >
> > > > Guido's blog (2009):
> > > >
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html
> > > <
> https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html>
> > > > Follow-up discussion (2024):
> > > > https://discuss.python.org/t/finishing-the-great-renaming/54082 <
> > > https://discuss.python.org/t/finishing-the-great-renaming/54082>
> > > >
> > > > Cheers,
> > > > Omkar
> > > >
> > > > On Wed, Oct 23, 2024 at 7:10 AM Wei Lee <weilee.rx@gmail.com
> <mailto:
> > > weilee.rx@gmail.com>> wrote:
> > > >
> > > > > I think we should probably just accept it as an airflow term. At
> least,
> > > > > thatâs how I understood it when I first used Airflow. I feel
> renaming
> > > it
> > > > at
> > > > > this stage would require considerable effort from maintainers and
> > > > existing
> > > > > users without providing equivalent benefits.
> > > > >
> > > > > Best,
> > > > > Wei
> > > > >
> > > > > > On Oct 23, 2024, at 8:01 AM, Kaxil Naik <kaxilnaik@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > > > >
> > > > > > Same agreed with Brent & Daniel -- maybe we re-kindle this
> discussion
> > > > for
> > > > > > Airflow 4 :) -- but right now it will cause too much disruption
> > > > > >
> > > > > > On Tue, 22 Oct 2024 at 21:27, Constance Martineau
> > > > > > <constance@astronomer.io.inva <mailto:
> constance@astronomer.io.inva>lid>
> > > wrote:
> > > > > >
> > > > > >> In my experience, when you ask those with Airflow experience
> what a
> > > > dag
> > > > > is,
> > > > > >> they'll start talking about workflow attributes - stuff like
> dags
> > > > being
> > > > > a
> > > > > >> series of steps or tasks with owners. The structure doesn't
> come up.
> > > > > >>
> > > > > >> Echo-ing others, at this point, my vote is to embrace the name
> and
> > > > > >> de-emphasize the mathematical structure aspect.
> > > > > >>
> > > > > >> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka
> > > > > <vikram@astronomer.io.inva <ma...@astronomer.io.inva>lid>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> It's an interesting discussion and I remember struggling with
> this
> > > > > when I
> > > > > >>> started working with Airflow.
> > > > > >>> But, I also agree with the viewpoint of it being an established
> > > > concept
> > > > > >> now
> > > > > >>> regardless of the origin.
> > > > > >>>
> > > > > >>> I am personally leaning towards the perspective best expressed
> by
> > > > > Daniel
> > > > > >>> Standish and Brent of using Dag as a word, rather than the
> computer
> > > > > >> science
> > > > > >>> concept.
> > > > > >>>
> > > > > >>> Best regards,
> > > > > >>> Vikram
> > > > > >>>
> > > > > >>>
> > > > > >>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko
> > > > > >> <onikolas@amazon.com.inva <ma...@amazon.com.inva>lid
> > > > > >>>>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> I agree with the general sentiment of: You're right Ryan, DAG
> > > isn't
> > > > > >> great
> > > > > >>>> and I'd rather workflow, but changing it will cause much more
> > > > wreckage
> > > > > >>> than
> > > > > >>>> it solves.
> > > > > >>>>
> > > > > >>>> Also agree with the idea to just move away from defining DAG.
> I
> > > > think
> > > > > >>>> we've been naturally doing that as a community for a while now
> > > > anyway,
> > > > > >> so
> > > > > >>>> that feels like a natural step.
> > > > > >>>>
> > > > > >>>> Cheers,
> > > > > >>>> Niko
> > > > > >>>>
> > > > > >>>> ________________________________
> > > > > >>>> From: Ash Berlin-Taylor <ash@apache.org <mailto:
> ash@apache.org>>
> > > > > >>>> Sent: Tuesday, October 22, 2024 9:06:39 AM
> > > > > >>>> To: dev@airflow.apache.org <ma...@airflow.apache.org>
> > > > > >>>> Subject: RE: [EXT] Airflow should deprecate the term "DAG"
> for end
> > > > > >> users
> > > > > >>>>
> > > > > >>>> CAUTION: This email originated from outside of the
> organization.
> > > Do
> > > > > not
> > > > > >>>> click links or open attachments unless you can confirm the
> sender
> > > > and
> > > > > >>> know
> > > > > >>>> the content is safe.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> AVERTISSEMENT: Ce courrier Ă©lectronique provient dâun
> expéditeur
> > > > > >> externe.
> > > > > >>>> Ne cliquez sur aucun lien et nâouvrez aucune piĂšce jointe si
> vous
> > > ne
> > > > > >>> pouvez
> > > > > >>>> pas confirmer lâidentitĂ© de lâexpĂ©diteur et si vous nâĂȘtes pas
> > > > certain
> > > > > >>> que
> > > > > >>>> le contenu ne présente aucun risque.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Best argument in favour of keeping âdagsâ as a term â getting
> to
> > > > > re-use
> > > > > >>>> puns like https://i.imgflip.com/1xhtwh.jpg <
> > > https://i.imgflip.com/1xhtwh.jpg>
> > > > > >>>>
> > > > > >>>> In all seriousness: I donât mind either way, both sides have
> good
> > > > > >> reasons
> > > > > >>>> presented.
> > > > > >>>>
> > > > > >>>> -a
> > > > > >>>>
> > > > > >>>>> On 22 Oct 2024, at 17:03, Daniel Standish
> > > > > >>>> <daniel.standish@astronomer.io.INVA <mailto:
> > > daniel.standish@astronomer.io.INVA>LID> wrote:
> > > > > >>>>>
> > > > > >>>>> Yeah just say, when asked where the name comes from, "well,
> no
> > > one
> > > > > >>>> actually
> > > > > >>>>> knows but..." and then make something up.
> > > > > >>>>>
> > > > > >>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk <
> jarek@potiuk.com
> > > <ma...@potiuk.com>>
> > > > > >> wrote:
> > > > > >>>>>
> > > > > >>>>>> Just to clarify - "directed acyclic graph" is the
> > > tongue-twister,
> > > > > >>>>>>
> > > > > >>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk <
> jarek@potiuk.com
> > > <ma...@potiuk.com>>
> > > > > >>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> I like what both Daniel and Brent wrote. I would very much
> want
> > > > to
> > > > > >> be
> > > > > >>>>>> able
> > > > > >>>>>>> to say just "dag" without explaining it further.
> > > > > >>>>>>>
> > > > > >>>>>>> For me every time I explain "DAG" at a talk it's a
> > > > tongue-twister,
> > > > > >>> and
> > > > > >>>> I
> > > > > >>>>>>> almost stutter on trying to recall how to pronounce it
> > > properly.
> > > > > >>>>>>>
> > > > > >>>>>>> J.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi
> > > > > >>>>>> <brent@astronomer.io.inva <mailto:brent@astronomer.io.inva
> >lid>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> I remember we explored renaming "DAG" when starting on
> AIP-38
> > > to
> > > > > >>>>>> modernize
> > > > > >>>>>>>> the UI. Both "pipeline" or "workflow" are more
> descriptive of
> > > > what
> > > > > >>> one
> > > > > >>>>>> is
> > > > > >>>>>>>> actually doing while Directed Acyclic Graph is an
> > > implementation
> > > > > >>>> detail.
> > > > > >>>>>>>> But I agree with Daniel Standish, at this point "DAG" has
> > > become
> > > > > >>> "dag"
> > > > > >>>>>> , a
> > > > > >>>>>>>> word in its own right.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Examples for "dag" are abound in community discussion,
> Airflow
> > > > > >>> Summit
> > > > > >>>>>>>> talks, documentation and even in the UI. Let's embrace
> "dag".
> > > A
> > > > > >> user
> > > > > >>>>>> just
> > > > > >>>>>>>> needs to learn one new word vs the technical concept
> behind
> > > that
> > > > > >>>> word. I
> > > > > >>>>>>>> think that is much less effort than refactoring so much
> code,
> > > > > >>>>>>>> documentation, blog posts, stack overflow questions, etc.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel Standish
> > > > > >>>>>>>> <daniel.standish@astronomer.io.inva <mailto:
> > > daniel.standish@astronomer.io.inva>lid> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> I am skeptical. Seems like introducing a lot of pain for
> > > > > >>>> questionable
> > > > > >>>>>>>>> benefit. But, I am def sympathetic to the idea. I agree
> the
> > > > > >>>>>>>> association
> > > > > >>>>>>>>> with "directed acyclic graph" is not helpful.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> And along those lines, I offer here some less invasive
> > > > > >> mitigations.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> One thing we can do no matter what is to de-emphasize the
> > > math
> > > > > >> nerd
> > > > > >>>>>>>> origins
> > > > > >>>>>>>>> of the name. That is to say, in docs / website / etc,
> *never
> > > > > >>> define*
> > > > > >>>>>>>>> airflow's "dag" concept as a directed acyclic graph.
> Always
> > > > > >> define
> > > > > >>>> it
> > > > > >>>>>>>> as a
> > > > > >>>>>>>>> pipeline, collection of tasks, workflow etc.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> The "directed acyclic graph" part of it is like a
> historical
> > > > > >>>> footnote,
> > > > > >>>>>>>> and
> > > > > >>>>>>>>> we could make one mention of it somewhere hidden.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> We could also start using lowercase in the docs in
> general
> > > e.g.
> > > > > >>>>>> writing
> > > > > >>>>>>>>> "dag" / "dags" instead of writing "DAG" / "DAGs" etc. The
> > > > upper
> > > > > >>> case
> > > > > >>>>>>>> part
> > > > > >>>>>>>>> of it makes it look like an acronym; but "dag" in
> airlfow is
> > > > just
> > > > > >>> an
> > > > > >>>>>>>>> airflow concept and the association with "DAGs" is not
> really
> > > > > >>>>>> unhelpful.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> In other words embrace that "dag" in airflow is its own
> > > thing,
> > > > is
> > > > > >>>>>>>>> *not* strictly
> > > > > >>>>>>>>> speaking a directed acyclic graph (which nobody knows
> about
> > > > > >>> anyway),
> > > > > >>>>>> and
> > > > > >>>>>>>>> tell them what it is in simple terms that normal people
> > > > > >> understand.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk <
> > > jarek@potiuk.com <ma...@potiuk.com>
> > > > >
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> DAG is so embedded into what we do that it will be
> extremely
> > > > > >>>>>>>> difficult to
> > > > > >>>>>>>>>> get rid of it completely. Also I think it will make a
> lot of
> > > > > >>>>>> "google"
> > > > > >>>>>>>>>> searches and "stack overflow" searches not finding the
> right
> > > > > >>>>>> answers.
> > > > > >>>>>>>>> This
> > > > > >>>>>>>>>> is one of the strengths of Airflow - besides the
> community
> > > and
> > > > > >>> ideas
> > > > > >>>>>>>> that
> > > > > >>>>>>>>>> Bernd mentioned - is the vast number of examples,
> problems
> > > and
> > > > > >>>>>>>> solutions
> > > > > >>>>>>>>>> you can so easily find (and we have to remember that
> all the
> > > > AI
> > > > > >>>>>>>> trained
> > > > > >>>>>>>>> on
> > > > > >>>>>>>>>> past data will be also rather poorly matching queries of
> > > > people.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I am not too attached to DAG. I could easily switch.
> And if
> > > we
> > > > > >> do
> > > > > >>> -
> > > > > >>>>>> I
> > > > > >>>>>>>>>> would be for using workflow or pipeline instead of
> `dag` if
> > > > not
> > > > > >>> the
> > > > > >>>>>>>> above
> > > > > >>>>>>>>>> reason, but I think I am here with Igor that it might
> cause
> > > > more
> > > > > >>>>>>>> problems
> > > > > >>>>>>>>>> than it solves.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> But I am not 100% against - if others will think it's a
> good
> > > > > >>> idea, I
> > > > > >>>>>>>> am
> > > > > >>>>>>>>> ok
> > > > > >>>>>>>>>> with it.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> J,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek Bhakat
> > > > > >>>>>>>>>> <abhishek.bhakat@astronomer.io.inva <mailto:
> > > abhishek.bhakat@astronomer.io.inva>lid> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Agreed that the word DAG makes very less sense to
> someone
> > > new
> > > > > >> to
> > > > > >>>>>>>>> workflow
> > > > > >>>>>>>>>>> orchestration. But it does also show the nature of
> being
> > > > > >> acyclic.
> > > > > >>>>>>>> Sure,
> > > > > >>>>>>>>>> as
> > > > > >>>>>>>>>>> Bas mentioned, there are ways to workaround it. Still,
> in
> > > my
> > > > > >>>>>>>> opinion,
> > > > > >>>>>>>>>> there
> > > > > >>>>>>>>>>> is generally no need for cyclic behavior in workflow
> > > > > >>>>>> orchestration.
> > > > > >>>>>>>>> Most
> > > > > >>>>>>>>>>> (*if
> > > > > >>>>>>>>>>> not all*) cases can be in some way can be covered
> using an
> > > > > >>> acyclic
> > > > > >>>>>>>>> manner
> > > > > >>>>>>>>>>> with multiple runs. Hence, the idempotency. So I would
> want
> > > > the
> > > > > >>>>>>>>> "acyclic"
> > > > > >>>>>>>>>>> word to stick.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Regards,
> > > > > >>>>>>>>>>> Avi
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM <
> > > bernd.stroehle@kosakya.de <ma...@kosakya.de>>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Brilliant, I am on the way to become an Airflow Fan;
> so
> > > many
> > > > > >> new
> > > > > >>>>>>>>> ideas.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> The Term DAG is misleading; it should be replaced by
> the
> > > > more
> > > > > >>>>>>>> general
> > > > > >>>>>>>>>>> Term
> > > > > >>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow (Petri) Net
> > > (AFN)
> > > > > >>>>>> (maybe
> > > > > >>>>>>>>>>> without
> > > > > >>>>>>>>>>>> a direction);
> > > > > >>>>>>>>>>>> and ... these Graphs should be stored in a Graph
> Database.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Every Node or Sup-Graph of an Airflow Graph (AFG)
> might be
> > > > > >>>>>>>> assigned
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>> an
> > > > > >>>>>>>>>>>> executable (Python-, Rust-, ... ) member of a library.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> A running Graph might have a different structure than
> a
> > > > > >>>>>>>> configuration
> > > > > >>>>>>>>>>>> Graph.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Forget that if you think it's bullshit.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Best Regards
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Bernd Ströhle
> > > > > >>>>>>>>>>>> M: +49 171 5357916
> > > > > >>>>>>>>>>>> E: bernd.stroehle@gmail.com <mailto:
> > > bernd.stroehle@gmail.com>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> -----Original Message-----
> > > > > >>>>>>>>>>>> From: Igor Kholopov <ikholopov@google.com.INVA
> <mailto:
> > > ikholopov@google.com.INVA>LID>
> > > > > >>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM
> > > > > >>>>>>>>>>>> To: dev@airflow.apache.org <mailto:
> dev@airflow.apache.org
> > > >
> > > > > >>>>>>>>>>>> Subject: Re: Airflow should deprecate the term "DAG"
> for
> > > end
> > > > > >>>>>> users
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Even though the term "DAG" is clearly suboptimal, it
> is
> > > part
> > > > > >> of
> > > > > >>>>>>>>> Airflow
> > > > > >>>>>>>>>>>> DAG definition interface at so many levels, that any
> > > attempt
> > > > > >> to
> > > > > >>>>>>>>> change
> > > > > >>>>>>>>>> it
> > > > > >>>>>>>>>>>> will only introduce more chaos, not reduce it. The
> only
> > > > thing
> > > > > >>>>>>>> that is
> > > > > >>>>>>>>>>> worse
> > > > > >>>>>>>>>>>> than a poorly chosen name in the code is when there
> are
> > > two
> > > > > >> ways
> > > > > >>>>>>>> to
> > > > > >>>>>>>>>>> define
> > > > > >>>>>>>>>>>> the same thing. Countless articles and tutorials will
> > > > suddenly
> > > > > >>>>>>>> become
> > > > > >>>>>>>>>>>> confusing as they all refer to workflows as "DAG"s.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> We are already at risk of scaring the users away with
> a
> > > > number
> > > > > >>>>>> of
> > > > > >>>>>>>>>>> breaking
> > > > > >>>>>>>>>>>> changes in Airflow 3, promising even more breaking
> changes
> > > > for
> > > > > >>>>>> the
> > > > > >>>>>>>>> most
> > > > > >>>>>>>>>>>> basic things is not something that people are looking
> for.
> > > > > >>>>>>>> Attempting
> > > > > >>>>>>>>>> to
> > > > > >>>>>>>>>>>> change the fundamental terms will be interpreted as an
> > > even
> > > > > >>>>>>>> stronger
> > > > > >>>>>>>>>>> signal
> > > > > >>>>>>>>>>>> of project immaturity.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Given that, I oppose the idea of changing the term in
> the
> > > > long
> > > > > >>>>>>>> run. I
> > > > > >>>>>>>>>>> even
> > > > > >>>>>>>>>>>> stricter oppose the idea of deprecating it in the DAG
> > > > > >> definition
> > > > > >>>>>>>>>>> interface.
> > > > > >>>>>>>>>>>> We better put our time and efforts in other places in
> > > > Airflow,
> > > > > >>>>>> of
> > > > > >>>>>>>>> which
> > > > > >>>>>>>>>>>> there are plenty.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Kind regards,
> > > > > >>>>>>>>>>>> Igor
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas Harenslak
> > > > > >>>>>>>>>> <bas@astronomer.io.inva <mailto:bas@astronomer.io.inva
> >lid
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Couple of thoughts:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> 1. The boundaries/properties of âDAGâ have already
> faded
> > > > over
> > > > > >>>>>>>> time,
> > > > > >>>>>>>>>>>>> for example there are now several ways to create
> cyclic
> > > > > >>>>>> graphs,
> > > > > >>>>>>>>> e.g.
> > > > > >>>>>>>>>>>>> using the @continuous schedule. I imagine these
> > > properties
> > > > > >>>>>>>>> vanishing
> > > > > >>>>>>>>>>>>> even more in the future, so from that perspective I
> > > support
> > > > > >>>>>>>>> changing
> > > > > >>>>>>>>>>>>> âDAG" to a more generic name.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> 2. How other orchestration frameworks do naming:
> > > > > >>>>>>>>>>>>> Dagster: pipeline
> > > > > >>>>>>>>>>>>> Prefect: flow
> > > > > >>>>>>>>>>>>> Flyte: workflow
> > > > > >>>>>>>>>>>>> Temporal: workflow
> > > > > >>>>>>>>>>>>> Kestra: flow
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> I think âworkflowâ is the most fitting name.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> 3. Given the large impact of this change, I suggest
> > > > defining
> > > > > >> a
> > > > > >>>>>>>>> clear
> > > > > >>>>>>>>>>>>> path forward. Would we first introduce the
> deprecation in
> > > > > >>>>>>>> Airflow
> > > > > >>>>>>>>> 3,
> > > > > >>>>>>>>>>>>> and remove âDAGâ in Airflow 4?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Bas
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil <neil4real@gmail.com
> > > <ma...@gmail.com>>
> > > > wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> I don't see a problem with the term DAG, especially
> when
> > > > > >>>>>> most
> > > > > >>>>>>>>> other
> > > > > >>>>>>>>>>>>>> platforms embrace the term wholeheartedly.
> > > > > >>>>>>>>>>>>>> I don't see anything intimidating or confusing
> about it
> > > at
> > > > > >>>>>>>> all,
> > > > > >>>>>>>>>>>>>> changing the term though would be fairly confusing
> to
> > > most
> > > > > >>>>>>>> users
> > > > > >>>>>>>>>> who
> > > > > >>>>>>>>>>>>>> have been
> > > > > >>>>>>>>>>>>> using
> > > > > >>>>>>>>>>>>>> the term for years.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping Chung
> > > > > >>>>>>>>>>>>>> <tp@astronomer.io.inva <mailto:
> tp@astronomer.io.inva
> > > >lid
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> I totally agree with doing away with the term DAG.
> The
> > > > only
> > > > > >>>>>>>>>> problem
> > > > > >>>>>>>>>>>>> (aside
> > > > > >>>>>>>>>>>>>>> from actually telling peopleâincluding myselfâto
> stop
> > > > using
> > > > > >>>>>>>> the
> > > > > >>>>>>>>>>>>>>> term)
> > > > > >>>>>>>>>>>>> is to
> > > > > >>>>>>>>>>>>>>> come up with a reasonable alternative.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> I canât recall who, but someone mentioned
> âworkflowâ is
> > > > not
> > > > > >>>>>>>> very
> > > > > >>>>>>>>>>>>> accurate
> > > > > >>>>>>>>>>>>>>> for Airflow. The term âdefinitionâ was proposed,
> but
> > > > itâs a
> > > > > >>>>>>>> bit
> > > > > >>>>>>>>>>>>>>> broad; I tried to use it in a few places and kept
> > > finding
> > > > > >>>>>>>> myself
> > > > > >>>>>>>>>>>>>>> doubting âwhat definition?â and wanting to clarify
> âDAG
> > > > > >>>>>>>>>> definitionâ
> > > > > >>>>>>>>>>>>>>> (defeating the purpose).
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> TP
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens Scheffler
> > > > > >>>>>>>>>>>>>>>> <j_scheffler@gmx.de.INVA <mailto:
> > > j_scheffler@gmx.de.INVA>LID>
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Hi Ryan,
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thanks for posting. I share the exactly same
> > > > observation,
> > > > > >>>>>>>> had a
> > > > > >>>>>>>>>>>>>>>> short
> > > > > >>>>>>>>>>>>>>> laight because the DAG question is always an
> > > introduction
> > > > > >>>>>> if
> > > > > >>>>>>>>>>>>>>> someone
> > > > > >>>>>>>>>>>>> joins
> > > > > >>>>>>>>>>>>>>> the party. I think a global renaming makes sense.
> > > > > >>>>>> Especially
> > > > > >>>>>>>>> when
> > > > > >>>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>> rename Dataset to Asset this is also a reasonable
> step.
> > > > > >>>>>>>> Concepts
> > > > > >>>>>>>>>>>>>>> still
> > > > > >>>>>>>>>>>>> can
> > > > > >>>>>>>>>>>>>>> stay the same.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> So I hope I donât need to join hiding below the
> desk
> > > > with
> > > > > >>>>>>>> you
> > > > > >>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>> +1
> > > > > >>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>> raising the discussion.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Technically we can still think if we keep details
> of
> > > > > >>>>>> python
> > > > > >>>>>>>>> names
> > > > > >>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>> same because the execution is still a DAG⊠but user
> > > > facing
> > > > > >>>>>> it
> > > > > >>>>>>>>> is a
> > > > > >>>>>>>>>>>>> workflow.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Jens
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Sent from my Smartphone
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan Hatter <
> > > > > >>>>>>>>>> ryan.hatter@astronomer.io <mailto:
> ryan.hatter@astronomer.io
> > > >
> > > > > >>>>>>>>>>>>> .invalid>
> > > > > >>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Everyone please sheathe your swords... at least
> for
> > > > now.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> The term "DAG" has very little meaning to Airflow
> > > > users.
> > > > > >>>>>>>>> Indeed,
> > > > > >>>>>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>> has
> > > > > >>>>>>>>>>>>>>>>> little meaning outside of some mathematicians and
> > > > > >>>>>> software
> > > > > >>>>>>>>>>>>>>>>> engineers
> > > > > >>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>> whom the properties of a DAG actually matter. For
> > > > someone
> > > > > >>>>>>>> new
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> data engineering or workflow orchestration, one
> of
> > > the
> > > > > >>>>>>>> first
> > > > > >>>>>>>>>>>>>>>>> questions they
> > > > > >>>>>>>>>>>>>>> will
> > > > > >>>>>>>>>>>>>>>>> likely have is, "what on earth is a DAG?" The
> answer
> > > is
> > > > > >>>>>>>> almost
> > > > > >>>>>>>>>>>>>>>>> always, "It's a directed acyclic graph. You don't
> > > need
> > > > to
> > > > > >>>>>>>>> worry
> > > > > >>>>>>>>>>>>>>>>> about what
> > > > > >>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>> means; it's just a term for your workflow."
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> The term "DAG" is problematic for at least a
> couple
> > > > > >>>>>>>> important
> > > > > >>>>>>>>>>>> reasons:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> *Complexity for New Users*: As mentioned above,
> "DAG"
> > > > is
> > > > > >>>>>>>>>>>>>>>>> unnecessarily intimidating and confusing. We want
> > > > Airflow
> > > > > >>>>>>>> to
> > > > > >>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>> approachable, and
> > > > > >>>>>>>>>>>>>>> using
> > > > > >>>>>>>>>>>>>>>>> technical jargon like "DAG" right off the bat
> creates
> > > > an
> > > > > >>>>>>>>> initial
> > > > > >>>>>>>>>>>>>>> barrier to
> > > > > >>>>>>>>>>>>>>>>> understanding.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow Concepts*:
> The
> > > DAG
> > > > > >>>>>> is
> > > > > >>>>>>>>> just
> > > > > >>>>>>>>>>>>>>>>> one component of an Airflow workflow. The
> workflow
> > > > > >>>>>> includes
> > > > > >>>>>>>>> its
> > > > > >>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen other
> > > parameters,
> > > > > >>>>>> and
> > > > > >>>>>>>>> other
> > > > > >>>>>>>>>>>>>>>>> metadata that
> > > > > >>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>> DAG component doesnât account for.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Consider the following from the Airflow homepage
> > > > > >>>>>>>>>>>>>>>>> <https://airflow.apache.org/> <
> > > https://airflow.apache.org/>>.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Apache AirflowÂź is a platform created by the
> > > community
> > > > to
> > > > > >>>>>>>>>>>>>>> programmatically
> > > > > >>>>>>>>>>>>>>>>> author, schedule and monitor workflows.
> > > > > >>>>>>>>>>>>>>>>> Then, if we look at the "What is Airflow?" docs
> page
> > > > > >>>>>>>>>>>>>>>>> <
> > > > > >>>>>>>>>>
> > > > > >>
> https://airflow.apache.org/docs/apache-airflow/stable/index.html <
> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html>
> > > > > >>>>>>>>>>>>>>>>>> ,
> > > > > >>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>> can
> > > > > >>>>>>>>>>>>>>>>> see that the docs explain what Airflow is without
> > > using
> > > > > >>>>>>>> "DAG."
> > > > > >>>>>>>>>>>>>>>>> It's
> > > > > >>>>>>>>>>>>>>> only in
> > > > > >>>>>>>>>>>>>>>>> the *workflow* Python code that the term is
> > > introduced
> > > > > >>>>>> out
> > > > > >>>>>>>> of
> > > > > >>>>>>>>>>>>>>>>> nowhere
> > > > > >>>>>>>>>>>>>>> as a
> > > > > >>>>>>>>>>>>>>>>> comment that awkwardly tries to explain it:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> # A DAG represents a workflow, a collection of
> tasks
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs in these
> > > > > >>>>>> introductions
> > > > > >>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't orchestrate
> DAGs;
> > > it
> > > > > >>>>>>>>>>> orchestrates
> > > > > >>>>>>>>>>>> workflows*.
> > > > > >>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>> DAG is the model that, for reasons irrelevant to
> > > almost
> > > > > >>>>>>>> every
> > > > > >>>>>>>>>>>>>>>>> user, workflows must adhere to.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> So, I propose at least adding an alias for the
> term
> > > > "DAG"
> > > > > >>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>> updating documentation to replace "DAG" with
> > > > "workflow".
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> For example, instead of...
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> @dag(
> > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > >>>>>>>>>>>>>>>>> ...
> > > > > >>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1)
> > > > > >>>>>>>>>>>>>>>>> )
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Users could do...
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> @workflow(
> > > > > >>>>>>>>>>>>>>>>> schedule="@daily",
> > > > > >>>>>>>>>>>>>>>>> ...
> > > > > >>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1)
> > > > > >>>>>>>>>>>>>>>>> )
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> And with that... I will start running away.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>
> ------------------------------------------------------------------
> > > > > >>>>>>>>>>>>>>>> --- To unsubscribe, e-mail:
> > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > > > >>>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > >>>>>>>> dev-help@airflow.apache.org <mailto:
> > > dev-help@airflow.apache.org>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>
> -------------------------------------------------------------------
> > > > > >>>>>>>>>>>>>>> -- To unsubscribe, e-mail:
> > > > > >>>>>>>> dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > > > >>>>>>>>>>>>>>> For additional commands, e-mail:
> > > > > >>>>>> dev-help@airflow.apache.org <mailto:
> dev-help@airflow.apache.org
> > > >
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>
> > > ---------------------------------------------------------------------
> > > > > >>>>>>>>>>>> To unsubscribe, e-mail:
> > > dev-unsubscribe@airflow.apache.org <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > > > >>>>>>>>>>>> For additional commands, e-mail:
> > > > dev-help@airflow.apache.org <ma...@airflow.apache.org>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> <mailto:
> > > dev-unsubscribe@airflow.apache.org>
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> <mailto:
> > > dev-help@airflow.apache.org>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>