You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Sumit Maheshwari <su...@gmail.com> on 2025/07/24 17:30:01 UTC
[DISCUSS] AIP-92 Isolate DAG parsing logic
Hello everyone,
I have created a draft version to separate the running of DAG processor
from Airflow's core services and moved it closer to the execution side.
Please review the proposal and provide your valuable feedback.
PS: We are in the process of adopting Airflow3 from some in-house setup, so
I might not have a full understanding of the latest Airflow concepts and
some other nitty-gritties.
Thanks,
Sumit
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
> However, the main issue we are trying to solve is not to provide
DB creds to the customer teams, who are using Airflow as a multi-tenant
orchestration platform. I've updated the doc to reflect this point as well.
Yep. We were waiting for someone like you/Uber who needs it to take the
lead on that if you need it for similar purposes. Glad you are taking it on.
> #2. Yeah, we would need something similar for triggerers as well, but that
can be done as part of a different AIP
You won't achieve your goal of "true" isolation of user code if you don't
do triggerer. I think if the goal is to achieve it - it should cover both.
J.
On Fri, Jul 25, 2025 at 11:49 AM Sumit Maheshwari <su...@gmail.com>
wrote:
> Thanks Jarek and Ash, for the initial review. It's good to know that the
> DAG processor has some preemptive measures in place to prevent access
> to the DB. However, the main issue we are trying to solve is not to provide
> DB creds to the customer teams, who are using Airflow as a multi-tenant
> orchestration platform. I've updated the doc to reflect this point as well.
>
> Answering Jarek's points,
>
> #1. Yeah, had forgot to write about token mechanism, added that in doc, but
> still how the token can be obtained (safely) is still open in my mind. I
> believe the token used by task executors can be created outside of it as
> well (I may be wrong here).
>
> #2. Yeah, we would need something similar for triggerers as well, but that
> can be done as part of a different AIP
>
> #3. Yeah, I also believe the API should work largely.
>
> #4. Added that in the AIP, that instead of dag_dirs we can work with
> dag_bundles and every dag-processor instance would be treated as a diff
> bundle.
>
> Also, added points around callbacks, as these are also fetched directly
> from the DB.
>
> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > A clarification to this - the dag parser today is likely not protection
> > against a dedicated malicious DAG author, but it does protect against
> > casual DB access attempts - the db session is blanked out in the parsing
> > process , as are the env var configs
> >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > -
> > is this perfect no? but it’s much more than no protection
> > Oh absolutely.. This is exactly what we discussed back then in March I
> > think - and the way we decided to go for 3.0 with full knowledge it's not
> > protecting against all threats.
> >
> > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > A clarification to this - the dag parser today is likely not protection
> > > against a dedicated malicious DAG author, but it does protect against
> > > casual DB access attempts - the db session is blanked out in the
> parsing
> > > process , as are the env var configs
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > - is this perfect no? but it’s much more than no protection
> > >
> > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > Currently in the DagFile processor there is no built-in protection
> > > against
> > > > user code from Dag Parsing to - for example - read database
> > > > credentials from airflow configuration and use them to talk to DB
> > > directly.
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
Just to add on that. We are still discussing details on the
Authentication/Tokens on that one. I think it's in a really good shape and
there are few details to work out - I will ask for a little more patience
on that one, from the security point of view. I added a few comments on the
doc - so Sumit, Igor - who discussed it there, please take a look - we can
discuss some of the things in the wikie page and I will want to make a
comprehensive "token/auth" document that will cover all (known) cases.
J.
On Thu, Aug 21, 2025 at 8:47 PM Kataria, Ramit <ra...@amazon.com.invalid>
wrote:
> I've been working on making Deadline callbacks work on the executor in
> addition to the Triggerer. That means introducing a new Workload type for
> the executor. I didn't want to tie it specifically to Deadlines and
> realized it would be a good idea to introduce generic Callbacks that the
> new workload can reference.
>
> Here's an initial PR for adding a new Callback table and refactoring
> existing Deadline callbacks that run on the Triggerer to use these instead:
> https://github.com/apache/airflow/pull/54796
>
> Ideally, once AIP-92 is in progress, these callbacks can be used or all
> the `on_*_callback`s as well. I want to ensure that these are generic
> enough for additional subclasses that would implement them. So, I'm looking
> for feedback/comments to align on a common definition for the new Workload
> type and the Callback model definition.
>
> Thanks,
> Ramit
>
>
> On 2025-08-13, 1:39 AM, "Jarek Potiuk" <jarek@potiuk.com <mailto:
> jarek@potiuk.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
>
>
>
> Those are all questions that we will open up for discussion once we get the
> basic foundation :)
>
>
> On Wed, Aug 13, 2025 at 10:16 AM Sumit Maheshwari <sumeet.manit@gmail.com
> <ma...@gmail.com>>
> wrote:
>
>
> > Awesome, having a standard approach for all kinds of authentication would
> > be great, looking forward to it.
> >
> > BTW, on a side note, I see that as of now, things like Connections,
> > Variables, and XComs which are present under Execution API namespace,
> don't
> > have any authorization model (left with TODOs), so is there any plan how
> > they will work, cause we might need something similar for
> > Dag-processor/Triggerer as well.
> >
> > Also, as we've decided to create a diff API namespace and not use
> execution
> > namespace for hosting new APIs required by dag-processor/triggerer, do
> > we've to copy these classes & routes OR atleast refactor them, so they
> can
> > serve all API namespaces.
> >
> > On Wed, Aug 13, 2025 at 1:27 PM Jarek Potiuk <jarek@potiuk.com <mailto:
> jarek@potiuk.com>> wrote:
> >
> > > Will do. We are also discussing - for now - within the security team -
> > the
> > > various aspects of authentication approach we want to have - for both
> > > "UI/User authentication" as well as "Long running services" and how
> they
> > > relate to token exchange, invalidation and other scenarios. What we
> will
> > > come up - I hope shortly - is a proposal of general "model" of all
> kinds
> > of
> > > security and authentication scenarios, so that we do not have to
> reinvent
> > > the wheel and try to figure out all the aspects with individual AIPs.
> We
> > > are not far from bringing it to the open discussion at devlist, but we
> > have
> > > to be careful about some of the aspects that we might need to improve
> in
> > > the current setup to close some - small - loopholes so bear with us :)
> > >
> > > On Wed, Aug 13, 2025 at 9:33 AM Sumit Maheshwari <
> sumeet.manit@gmail.com <ma...@gmail.com>
> > >
> > > wrote:
> > >
> > > > Thanks Ash and Jarek, for the detailed comments. I largely agree with
> > the
> > > > points mentioned by you guys, hence I updated the AIP and added a
> > section
> > > > on Authentication, API Versioning, and Packaging as well. Please go
> > > through
> > > > it once more and let me know if there are more things to consider
> > before
> > > I
> > > > open it for voting.
> > > >
> > > >
> > > > On Thu, Aug 7, 2025 at 6:03 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>> wrote:
> > > >
> > > > > Also
> > > > >
> > > > > > 1. The Authentication token. How will this long lived token work
> > > > without
> > > > > being insecure. Who and what will generate it? How will we identify
> > > > > top-level requests for Variables in order to be able to add
> Variable
> > > > > RBAC/ACLs. This is an important enough thing that I think it needs
> > > > > discussion before we vote on this AIP.
> > > > >
> > > > > We are currently discussing - in the security team - approach for
> JWT
> > > > token
> > > > > handling, so likely we could move the discussion there, it does
> have
> > > some
> > > > > security implications and I think we should bring our finding to
> the
> > > > > devlist when we complete it, but I think we should add this case
> > there.
> > > > > IMHO we should have a different approach for UI, different for
> Tasks,
> > > > > different for Triggerer, and different for DagProcessor. (possibly
> > the
> > > > > Trigerer and DagProcessor could be the same because they share
> > > > essentially
> > > > > the same long-living token. Ash - I will add this to the discussion
> > > > there.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>>
> > wrote:
> > > > >
> > > > > > Ah.. So if we are talking about a more complete approach - seeing
> > > those
> > > > > > comments from Ash - make me think if we should have another AIP.
> > > > > > (connected) about splitting the distributions. We have never
> > > finalized
> > > > it
> > > > > > (nor even discussed id) but Ash - you had some initial document
> for
> > > > that.
> > > > > > So maybe we should finalize it and rather than specify it in this
> > > AIP -
> > > > > > have a separate AIP about distribution split that AIP-92 could
> > depend
> > > > on.
> > > > > > It seems much more reasonable to split "distribution and code
> > split"
> > > > from
> > > > > > parsing isolation I think and implement them separately/in
> > parallel.
> > > > > >
> > > > > > Reading Ash comments (and maybe I am going a bit further than
> Ash)
> > it
> > > > > > calls for something that I am a big proponent of - splitting
> > > > > "airflow-core"
> > > > > > and having a different "scheduler". "webserver", "dag processor"
> > and
> > > > > > "triggerer" distributions. Now - we have the capability of having
> > > > > "shared"
> > > > > > code - we do not need "common" code to make it happen - because
> we
> > > can
> > > > > > share code.
> > > > > >
> > > > > > What it could give us - on top of clean client/server split, we
> > could
> > > > > have
> > > > > > different dependencies used by those distributions. Additionally,
> > we
> > > > > could
> > > > > > also split-off the executors from providers and finally implement
> > it
> > > in
> > > > > the
> > > > > > way that scheduler does not use providers at all (not even
> > > > > cncf.kubernetes
> > > > > > nor celery providers installed in scheduler nor webserver but
> > > > "executors"
> > > > > > packages instead. The code sharing approach with symlinks we have
> > now
> > > > > will
> > > > > > make it a .... breeze :) . That would also imply sharing
> > "connection"
> > > > > > definitions through DB, and likely implementing "test connection"
> > > > > > feature properly finally (i.e executing test connection in
> worker /
> > > > > > triggerer rather than in web server which is a reason why we
> > disabled
> > > > it
> > > > > by
> > > > > > default now). This way "api-server" would not need any of the
> > > providers
> > > > > to
> > > > > > be installed either which IMHO is the biggest win from a security
> > > point
> > > > > of
> > > > > > view.
> > > > > >
> > > > > > And the nice thing about it is that it would be rather
> transparent
> > > when
> > > > > > anyone uses "pip install apache-airflow" - it would behave
> exactly
> > > the
> > > > > > same, no more complexity involved, simply more distributions
> > > installed
> > > > > when
> > > > > > 'apache-airflow" meta-distribution is used, but it would allow
> > those
> > > > who
> > > > > > want to implement a more complex and secure setup to have
> different
> > > > > > "environments" with modularized pieces of airflow installed -
> only
> > > > > > "apache-airflow-dag-processor + task-sdk + providers" where
> > > > dag-processor
> > > > > > is run, only "apache-airflow-scheduler + executors" where
> scheduler
> > > is
> > > > > > installed only "apache-airflow-task-sdk + providers" where
> workers
> > > are
> > > > > > running, only "apache-airflow-api-server" where api-server is
> > running
> > > > and
> > > > > > only "apache-airflow-trigger + task-sdk + providers" .
> > > > > >
> > > > > > I am happy (Ash If you are fine with that) to take that original
> > > > document
> > > > > > over and lead this part and new AIP to completion (including
> > > > > > implementation), I am very much convinced that this will lead to
> > much
> > > > > > better dependency security and more modular code without
> impacting
> > > the
> > > > > > "apache-airflow" installation complexity.
> > > > > >
> > > > > > If we do it this way- the part of code/clean split would be
> > > "delegated
> > > > > > out" from AIP-92 to this new AIP and turned into dependency.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <ash@apache.org
> <ma...@apache.org>>
> > > > wrote:
> > > > > >
> > > > > >> This AIP is definitely heading in the right direction and is a
> > > feature
> > > > > >> I’d like to see.
> > > > > >>
> > > > > >> For me the outstanding things that need more detail:
> > > > > >>
> > > > > >> 1. The Authentication token. How will this long lived token work
> > > > without
> > > > > >> being insecure. Who and what will generate it? How will we
> > identify
> > > > > >> top-level requests for Variables in order to be able to add
> > Variable
> > > > > >> RBAC/ACLs. This is an important enough thing that I think it
> needs
> > > > > >> discussion before we vote on this AIP.
> > > > > >> 2. Security generally — how will this work, especially with the
> > > > > >> multi-team? I think this likely means making the APIs work on
> the
> > > > bundle
> > > > > >> level as you mention in the doc, but I haven’t thought deeply
> > about
> > > > this
> > > > > >> yet.
> > > > > >> 3. API Versioning? One of the the key driving goals with AIP-72
> > and
> > > > the
> > > > > >> Task Execution SDK was the idea that “you can upgrade the API
> > server
> > > > as
> > > > > you
> > > > > >> like, and your clients/workers never need to work” — i.e. the
> API
> > > > > server is
> > > > > >> 100% working with all older versions of the TaskSDK. I don’t
> know
> > if
> > > > we
> > > > > >> will achieve that goal in the long run but it is the desire, and
> > > part
> > > > of
> > > > > >> why we are using CalVer and the Cadwyn library to provide API
> > > > > versioning.
> > > > > >> 4. As mentioned previously, not sure the existing serialised
> JSON
> > > > format
> > > > > >> for DAGs is correct, but since that now has version and we
> already
> > > > have
> > > > > the
> > > > > >> ability to upgrade that somewhere in the Airflow Core that
> doesn’t
> > > > > >> necessarily become a blocker/pre-requisite for this AIP.
> > > > > >>
> > > > > >> I think Dag parsing API client+submission+parsing process
> manager
> > > > should
> > > > > >> either live in the Task SDK dist, or in a new separate dist that
> > > uses
> > > > > >> TaskSDK, but crucially not in apache-airflow-core. My reason for
> > > this
> > > > is
> > > > > >> that I want it to be possible for the server components
> > (scheduler,
> > > > API
> > > > > >> server) to not need task-sdk installed (just for
> > > cleanliness/avoiding
> > > > > >> confusion about what versions it needs) and also vice-verse, to
> be
> > > > able
> > > > > to
> > > > > >> run a “team worker bundle” (Dag parsing, workers,
> triggered/async
> > > > > workers)
> > > > > >> on whatever version of TaskSDK they choose, again without
> > > > > >> apache-airflow-core installed for avoidance of doubt.
> > > > > >>
> > > > > >> Generally I would like this as it means we can have a nicer
> > > separation
> > > > > of
> > > > > >> Core and Dag parsing code, as the dag parsing itself uses the
> SDK,
> > > it
> > > > > would
> > > > > >> be nice to have a proper server/client split, both from a
> tighter
> > > > > security
> > > > > >> point-of-view, but also from a code layout point of view.
> > > > > >>
> > > > > >> -ash
> > > > > >>
> > > > > >>
> > > > > >> > On 7 Aug 2025, at 12:36, Jarek Potiuk <jarek@potiuk.com
> <ma...@potiuk.com>> wrote:
> > > > > >> >
> > > > > >> > Well, you started it - so it's up to you to decide if you
> think
> > we
> > > > > have
> > > > > >> > consensus, or whether we need a vote.
> > > > > >> >
> > > > > >> > And It's not a question of "informal" vote but it's rather
> clear
> > > > > >> following
> > > > > >> > the https://www.apache.org/foundation/voting.html <
> https://www.apache.org/foundation/voting.html> that we
> > either
> > > > > need a
> > > > > >> > LAZY CONSENSUS or VOTE thread. Both are formal.
> > > > > >> >
> > > > > >> > This is the difficult part when you have a proposal, to assess
> > (by
> > > > > you)
> > > > > >> > whether we are converging to consensus or whether vote is
> > needed.
> > > > > There
> > > > > >> is
> > > > > >> > no other body or "authority" to do it for you.
> > > > > >> >
> > > > > >> > J.
> > > > > >> >
> > > > > >> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <
> > > > > sumeet.manit@gmail.com <ma...@gmail.com>
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Sorry for nudging again, but can we get into some consensus
> on
> > > > this?
> > > > > I
> > > > > >> mean
> > > > > >> >> if this AIP isn't good enough, then we can drop it altogether
> > and
> > > > > >> someone
> > > > > >> >> can rethink the whole thing. Should we do some kind of
> informal
> > > > > voting
> > > > > >> and
> > > > > >> >> close this thread?
> > > > > >> >>
> > > > > >> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <
> jarek@potiuk.com <ma...@potiuk.com>>
> > > > > wrote:
> > > > > >> >>
> > > > > >> >>>>> My main concern with this right now is the serialisation
> > > format
> > > > of
> > > > > >> >> DAGs
> > > > > >> >>> —
> > > > > >> >>>> it wasn’t really designed with remote submission in mind,
> so
> > it
> > > > > need
> > > > > >> >> some
> > > > > >> >>>> careful examination to see if it is fit for this purpose or
> > > not.
> > > > > >> >>>
> > > > > >> >>> I understand Ash's concerns - the format has not been
> designed
> > > > with
> > > > > >> >>> size/speed optimization in mind so **possibly** we could
> > design
> > > a
> > > > > >> >> different
> > > > > >> >>> format that would be better suited.
> > > > > >> >>>
> > > > > >> >>> BUT ... Done is better than perfect.
> > > > > >> >>>
> > > > > >> >>> I think there are a number of risks involved in changing the
> > > > format
> > > > > >> and
> > > > > >> >> it
> > > > > >> >>> could significantly increase time of development with
> > uncertain
> > > > > gains
> > > > > >> at
> > > > > >> >>> the end - also because of the progress in compression that
> > > > happened
> > > > > >> over
> > > > > >> >>> the last few years.
> > > > > >> >>>
> > > > > >> >>> It might be a good idea to experiment a bit with different
> > > > > compression
> > > > > >> >>> algorithms for "our" dag representation and possibly we
> could
> > > find
> > > > > the
> > > > > >> >> best
> > > > > >> >>> algorithm for "airflow dag" type of json data. There are a
> lot
> > > of
> > > > > >> >>> repetitions in the JSON representation and I guess in "our"
> > json
> > > > > >> >>> representation there are some artifacts and repeated
> sections
> > > that
> > > > > >> simply
> > > > > >> >>> might compress well with different algorithms. Also in this
> > case
> > > > > >> >>> speed matters (and CPU trade-off).
> > > > > >> >>>
> > > > > >> >>> Looking at compression "theory" - before we experiment with
> > it -
> > > > > >> there is
> > > > > >> >>> the relatively new standard "zstandard"
> > > > > >> https://github.com/facebook/zstd <
> https://github.com/facebook/zstd>
> > > > > >> >>> compression opensourced in 2016 which I've heard good things
> > > > about -
> > > > > >> >>> especially that it maintains a very good compression rate
> for
> > > text
> > > > > >> data,
> > > > > >> >>> but also it is tremendously fast - especially for
> > decompression
> > > > > >> (which is
> > > > > >> >>> super important factor for us - we compress new DAG
> > > representation
> > > > > far
> > > > > >> >> less
> > > > > >> >>> often than decompress it in general case). It is
> standardized
> > in
> > > > RFC
> > > > > >> >>> https://datatracker.ietf.org/doc/html/rfc8878 <
> https://datatracker.ietf.org/doc/html/rfc8878> and there are
> > > > various
> > > > > >> >>> implementations and it is even being added to Python
> standard
> > > > > library
> > > > > >> in
> > > > > >> >>> Python 3.14
> > > > > >> https://docs.python.org/3.14/library/compression.zstd.html <
> https://docs.python.org/3.14/library/compression.zstd.html>
> > > > > >> >> and
> > > > > >> >>> there is a very well maintained python binding library
> > > > > >> >>> https://pypi.org/project/zstd/ <
> https://pypi.org/project/zstd/> to Yann Collet (algorithm
> > > author)
> > > > > >> ZSTD C
> > > > > >> >>> library. And libzstd is already part of our images - it is
> > > needed
> > > > by
> > > > > >> >> other
> > > > > >> >>> dependencies of ours. All with BSD licence, directly usable
> by
> > > us.
> > > > > >> >>>
> > > > > >> >>> I think this one might be a good candidate for us to try,
> and
> > > > > possibly
> > > > > >> >> with
> > > > > >> >>> zstd we could achieve both size and CPU overhead that would
> be
> > > > > >> comparable
> > > > > >> >>> with any "new" format we could come up with - especially
> that
> > we
> > > > are
> > > > > >> >>> talking merely about processing a huge blob between
> "storable"
> > > > > >> >> (compressed)
> > > > > >> >>> and "locally usable" state (Python dict). We could likely
> use
> > a
> > > > > >> streaming
> > > > > >> >>> JSON library (say the one that is used in Pydantic
> internally
> > > > > >> >>> https://github.com/pydantic/jiter <
> https://github.com/pydantic/jiter> - we already have it as
> > part
> > > of
> > > > > >> >>> Pydantic)
> > > > > >> >>> to also save memory - we could stream decompressed stream
> into
> > > > > jitter
> > > > > >> so
> > > > > >> >>> that both the json dict and string representation does not
> > have
> > > to
> > > > > be
> > > > > >> >>> loaded fully in memory at the same time. There are likely
> lots
> > > of
> > > > > >> >>> optimisations we could do - I mentioned possibly streaming
> the
> > > > data
> > > > > >> from
> > > > > >> >>> API directly to DB (if this is possible - not sure)
> > > > > >> >>>
> > > > > >> >>> J.
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> > > > > >> sumeet.manit@gmail.com <ma...@gmail.com>>
> > > > > >> >>> wrote:
> > > > > >> >>>
> > > > > >> >>>>>
> > > > > >> >>>>> My main concern with this right now is the serialisation
> > > format
> > > > of
> > > > > >> >>> DAGs —
> > > > > >> >>>>> it wasn’t really designed with remote submission in mind,
> so
> > > it
> > > > > need
> > > > > >> >>> some
> > > > > >> >>>>> careful examination to see if it is fit for this purpose
> or
> > > not.
> > > > > >> >>>>>
> > > > > >> >>>>
> > > > > >> >>>> I'm not sure on this point, cause if we are able to
> convert a
> > > DAG
> > > > > >> into
> > > > > >> >>>> JSON, then it has to be transferable over the internet.
> > > > > >> >>>>
> > > > > >> >>>> In particular One of the things I worry about is that the
> > JSON
> > > > can
> > > > > >> get
> > > > > >> >>> huge
> > > > > >> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> > > > > >> >>>>
> > > > > >> >>>>
> > > > > >> >>>> Yeah, agree on this, thats why we can transfer compressed
> > data
> > > > > >> instead
> > > > > >> >> of
> > > > > >> >>>> real json. Of course, this won't guarantee that the payload
> > > will
> > > > > >> always
> > > > > >> >>> be
> > > > > >> >>>> small enough, but we can't say that it'll definitely happen
> > > > either.
> > > > > >> >>>>
> > > > > >> >>>> I also wonder if as part of this proposal we should move
> the
> > > > > Callback
> > > > > >> >>>>> requests off the dag parsers and on to the workers instead
> > > > > >> >>>>
> > > > > >> >>>> let's make such a "workfload" implementation stream that
> > could
> > > > > >> support
> > > > > >> >>> both
> > > > > >> >>>>> - Deadlines and DAG parsing logic
> > > > > >> >>>>
> > > > > >> >>>>
> > > > > >> >>>> I don't have any strong opinion here, but it feels like
> it's
> > > > gonna
> > > > > >> blow
> > > > > >> >>> up
> > > > > >> >>>> the scope of the AIP too much.
> > > > > >> >>>>
> > > > > >> >>>>
> > > > > >> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <
> > jarek@potiuk.com <ma...@potiuk.com>>
> > > > > >> wrote:
> > > > > >> >>>>
> > > > > >> >>>>>> My main concern with this right now is the serialisation
> > > format
> > > > > of
> > > > > >> >>>> DAGs —
> > > > > >> >>>>> it wasn’t really designed with remote submission in mind,
> so
> > > it
> > > > > need
> > > > > >> >>> some
> > > > > >> >>>>> careful examination to see if it is fit for this purpose
> or
> > > not.
> > > > > >> >>>>>
> > > > > >> >>>>> Yep. That might be potentially a problem (or at least
> "need
> > > more
> > > > > >> >>>> resources
> > > > > >> >>>>> to run airflow") and that is where my "2x memory" came
> from
> > if
> > > > we
> > > > > do
> > > > > >> >> it
> > > > > >> >>>> in
> > > > > >> >>>>> a trivial way. Currently we a) keep the whole DAG in
> memory
> > > when
> > > > > >> >>>>> serializing it b) submit it to database (also using
> > > essentially
> > > > > some
> > > > > >> >>> kind
> > > > > >> >>>>> of API (implemented by the database client) - so we know
> the
> > > > whole
> > > > > >> >>> thing
> > > > > >> >>>>> "might work" but indeed if you use a trivial
> implementation
> > of
> > > > > >> >>> submitting
> > > > > >> >>>>> the whole json - it basically means that the whole json
> will
> > > > have
> > > > > to
> > > > > >> >>> also
> > > > > >> >>>>> be kept in the memory of API server. But we also compress
> it
> > > > when
> > > > > >> >>> needed
> > > > > >> >>>> -
> > > > > >> >>>>> I wonder what are the compression ratios we saw with those
> > > > > 10-20MBs
> > > > > >> >>> Dags
> > > > > >> >>>> -
> > > > > >> >>>>> if the problem is using strings where bool would suffice,
> > > > > >> compression
> > > > > >> >>>>> should generally help a lot. We could only ever send
> > > compressed
> > > > > data
> > > > > >> >>> over
> > > > > >> >>>>> the API - there seems to be no need to send "plain JSON"
> > data
> > > > over
> > > > > >> >> the
> > > > > >> >>>> API
> > > > > >> >>>>> or storing the plain JSON in the DB (of course that trades
> > > > memory
> > > > > >> for
> > > > > >> >>>> CPU).
> > > > > >> >>>>>
> > > > > >> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres)
> > have
> > > > > >> >> support
> > > > > >> >>>> for
> > > > > >> >>>>> any kind if binary data streaming - because that could
> help
> > a
> > > > lot
> > > > > of
> > > > > >> >> if
> > > > > >> >>>> we
> > > > > >> >>>>> could use streaming HTTP API and chunk and append the
> binary
> > > > > chunks
> > > > > >> >>> (when
> > > > > >> >>>>> writing) - or read data in chunks ans stream them back via
> > the
> > > > > API.
> > > > > >> >>> That
> > > > > >> >>>>> could seriously decrease the amount of memory needed by
> the
> > > API
> > > > > >> >> server
> > > > > >> >>> to
> > > > > >> >>>>> process such huge serialized dags.
> > > > > >> >>>>>
> > > > > >> >>>>> And yeah - I would also love the "execute task" to be
> > > > implemented
> > > > > >> >> here
> > > > > >> >>> -
> > > > > >> >>>>> but I am not sure if this should be part of the same
> effort
> > or
> > > > > maybe
> > > > > >> >> a
> > > > > >> >>>>> separate implementation? That sounds very loosely coupled
> > with
> > > > DB
> > > > > >> >>>>> isolation. And it seems a common theme - I think that
> would
> > > also
> > > > > >> make
> > > > > >> >>> the
> > > > > >> >>>>> sync Deadline alerts case that we discussed at the dev
> call
> > > > > today. I
> > > > > >> >>>> wonder
> > > > > >> >>>>> if that should not be kind of parallel (let's make such a
> > > > > >> "workfload"
> > > > > >> >>>>> implementation stream that could support both - Deadlines
> > and
> > > > DAG
> > > > > >> >>> parsing
> > > > > >> >>>>> logic. We have already two "users" for it and I really
> love
> > > the
> > > > > >> >> saying
> > > > > >> >>>> "if
> > > > > >> >>>>> you want to make something reusable - make it usable
> > first" -
> > > > > seems
> > > > > >> >>> like
> > > > > >> >>>>> we might have good opportunity to make such workload
> > > > > implementation
> > > > > >> >>>> "doubly
> > > > > >> >>>>> used" from the beginning which would increase chances it
> > will
> > > > be
> > > > > >> >>>>> "reusable" for other things as well :).
> > > > > >> >>>>>
> > > > > >> >>>>> J.
> > > > > >> >>>>>
> > > > > >> >>>>>
> > > > > >> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <
> > > > > ash@apache.org <ma...@apache.org>>
> > > > > >> >>>> wrote:
> > > > > >> >>>>>
> > > > > >> >>>>>> My main concern with this right now is the serialisation
> > > format
> > > > > of
> > > > > >> >>>> DAGs —
> > > > > >> >>>>>> it wasn’t really designed with remote submission in mind,
> > so
> > > it
> > > > > >> >> need
> > > > > >> >>>> some
> > > > > >> >>>>>> careful examination to see if it is fit for this purpose
> or
> > > > not.
> > > > > >> >>>>>>
> > > > > >> >>>>>> In particular One of the things I worry about is that the
> > > JSON
> > > > > can
> > > > > >> >>> get
> > > > > >> >>>>>> huge — I’ve seem this as large as 10-20Mb for some
> dags(!!)
> > > > > (which
> > > > > >> >> is
> > > > > >> >>>>>> likely due to things being included as text when a bool
> > might
> > > > > >> >>> suffice,
> > > > > >> >>>>> for
> > > > > >> >>>>>> example) But I don’t think “just submit the existing JSON
> > > over
> > > > an
> > > > > >> >>> API”
> > > > > >> >>>>> is a
> > > > > >> >>>>>> good idea.
> > > > > >> >>>>>>
> > > > > >> >>>>>> I also wonder if as part of this proposal we should move
> > the
> > > > > >> >> Callback
> > > > > >> >>>>>> requests off the dag parsers and on to the workers
> instead
> > —
> > > in
> > > > > >> >>> AIP-72
> > > > > >> >>>> we
> > > > > >> >>>>>> introduced the concept of a Workload, with the only one
> > > > existing
> > > > > >> >>> right
> > > > > >> >>>>> now
> > > > > >> >>>>>> is “ExecuteTask”
> > > > > >> >>>>>>
> > > > > >> >>>>>
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> <
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> >
> > > > > >> >>>>>> — it might be time to finally move task and dag callbacks
> > to
> > > > the
> > > > > >> >> same
> > > > > >> >>>>> thing
> > > > > >> >>>>>> and make dag parsers only responsible for, well, parsing.
> > :)
> > > > > >> >>>>>>
> > > > > >> >>>>>> These are all solvable problems, and this will be a great
> > > > feature
> > > > > >> >> to
> > > > > >> >>>>> have,
> > > > > >> >>>>>> but we need to do some more thinking and planning first.
> > > > > >> >>>>>>
> > > > > >> >>>>>> -ash
> > > > > >> >>>>>>
> > > > > >> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> > > > > >> >> sumeet.manit@gmail.com <ma...@gmail.com>
> > > > > >> >>>>
> > > > > >> >>>>>> wrote:
> > > > > >> >>>>>>>
> > > > > >> >>>>>>> Gentle reminder for everyone to review the proposal.
> > > > > >> >>>>>>>
> > > > > >> >>>>>>> Updated link:
> > > > > >> >>>>>>>
> > > > > >> >>>>>>
> > > > > >> >>>>>
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> >
> > > > > >> >>>>>>>
> > > > > >> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > > > > >> >>>>> sumeet.manit@gmail.com <ma...@gmail.com>
> > > > > >> >>>>>>>
> > > > > >> >>>>>>> wrote:
> > > > > >> >>>>>>>
> > > > > >> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and
> > others
> > > > > >> >>>>> suggested, I
> > > > > >> >>>>>>>> expanded the scope of this AIP and divided it into
> three
> > > > > phases.
> > > > > >> >>>> With
> > > > > >> >>>>>> the
> > > > > >> >>>>>>>> increased scope, the boundary line between this AIP and
> > > > AIP-85
> > > > > >> >>> got a
> > > > > >> >>>>>> little
> > > > > >> >>>>>>>> thinner, but I believe these are still two different
> > > > > >> >> enhancements
> > > > > >> >>> to
> > > > > >> >>>>>> make.
> > > > > >> >>>>>>>>
> > > > > >> >>>>>>>>
> > > > > >> >>>>>>>>
> > > > > >> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > > > >> >>>>>> sumeet.manit@gmail.com <ma...@gmail.com>>
> > > > > >> >>>>>>>> wrote:
> > > > > >> >>>>>>>>
> > > > > >> >>>>>>>>> Yeah, overall it makes sense to include Triggers as
> well
> > > to
> > > > be
> > > > > >> >>> part
> > > > > >> >>>>> of
> > > > > >> >>>>>>>>> this AIP and phase out the implementation. Though I
> > didn't
> > > > > >> >>> exclude
> > > > > >> >>>>>> Triggers
> > > > > >> >>>>>>>>> because "Uber" doesn't need that, I just thought of
> > > keeping
> > > > > the
> > > > > >> >>>> scope
> > > > > >> >>>>>> of
> > > > > >> >>>>>>>>> development small and achieving them, just like it was
> > > done
> > > > in
> > > > > >> >>>>> Airlfow
> > > > > >> >>>>>> 3 by
> > > > > >> >>>>>>>>> secluding only workers and not DAG-processor &
> Triggers.
> > > > > >> >>>>>>>>>
> > > > > >> >>>>>>>>> But if you think Triggers should be part of this AIP
> > > itself,
> > > > > >> >>> then I
> > > > > >> >>>>> can
> > > > > >> >>>>>>>>> do that and include Triggers as well in it.
> > > > > >> >>>>>>>>>
> > > > > >> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <
> > > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > > >> >>>
> > > > > >> >>>>> wrote:
> > > > > >> >>>>>>>>>
> > > > > >> >>>>>>>>>> I would very much prefer the architectural choices of
> > > this
> > > > > AIP
> > > > > >> >>> are
> > > > > >> >>>>>> based
> > > > > >> >>>>>>>>>> on
> > > > > >> >>>>>>>>>> "general public" needs rather than "Uber needs" even
> if
> > > > Uber
> > > > > >> >>> will
> > > > > >> >>>> be
> > > > > >> >>>>>>>>>> implementing it - so from my point of view having
> > Trigger
> > > > > >> >>>> separation
> > > > > >> >>>>>> as
> > > > > >> >>>>>>>>>> part of it is quite important.
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>> But that's not even this.
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>> We've been discussing for example for Deadlines
> (being
> > > > > >> >>> implemented
> > > > > >> >>>>> by
> > > > > >> >>>>>>>>>> Dennis and Ramit a possibility of short,
> > > > notification-style
> > > > > >> >>>>>> "deadlines"
> > > > > >> >>>>>>>>>> to be send to triggerer for execution - this is well
> > > > advanced
> > > > > >> >>> now,
> > > > > >> >>>>> and
> > > > > >> >>>>>>>>>> whether you want it or not Dag-provided code might be
> > > > > >> >> serialized
> > > > > >> >>>> and
> > > > > >> >>>>>> sent
> > > > > >> >>>>>>>>>> to triggerer for execution. This is part of our
> > "broader"
> > > > > >> >>>>>> architectural
> > > > > >> >>>>>>>>>> change where we treat "workers" and "triggerer"
> > similarly
> > > > as
> > > > > a
> > > > > >> >>>>> general
> > > > > >> >>>>>>>>>> executors of "sync" and "async" tasks respectively.
> > > That's
> > > > > >> >> where
> > > > > >> >>>>>> Airflow
> > > > > >> >>>>>>>>>> is
> > > > > >> >>>>>>>>>> evolving towards - inevitably.
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>> But we can of course phase things in out for
> > > > implementation -
> > > > > >> >>> even
> > > > > >> >>>>> if
> > > > > >> >>>>>> AIP
> > > > > >> >>>>>>>>>> should cover both, I think if the goal of the AIP and
> > > > > preamble
> > > > > >> >>> is
> > > > > >> >>>>>> about
> > > > > >> >>>>>>>>>> separating "user code" from "database" as the main
> > > reason,
> > > > it
> > > > > >> >>> also
> > > > > >> >>>>>> means
> > > > > >> >>>>>>>>>> Triggerer if you ask me (from design point of view at
> > > > least).
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>> Again implementation can be phased and even different
> > > > people
> > > > > >> >> and
> > > > > >> >>>>> teams
> > > > > >> >>>>>>>>>> might work on those phases/pieces.
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>> J.
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > > > >> >>>>>> sumeet.manit@gmail.com <ma...@gmail.com>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>> wrote:
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> > > triggerers
> > > > > as
> > > > > >> >>>> well,
> > > > > >> >>>>>>>>>> but
> > > > > >> >>>>>>>>>>>> that
> > > > > >> >>>>>>>>>>>> can be done as part of a different AIP
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>> You won't achieve your goal of "true" isolation of
> > user
> > > > code
> > > > > >> >> if
> > > > > >> >>>> you
> > > > > >> >>>>>>>>>> don't
> > > > > >> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it
> -
> > it
> > > > > >> >> should
> > > > > >> >>>>> cover
> > > > > >> >>>>>>>>>>> both.
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>> My bad, should've explained our architecture for
> > > triggers
> > > > as
> > > > > >> >>>> well,
> > > > > >> >>>>>>>>>>> apologies. So here it is:
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>> - Triggers would be running on a centralized
> service,
> > > so
> > > > > >> >> all
> > > > > >> >>>> the
> > > > > >> >>>>>>>>>> Trigger
> > > > > >> >>>>>>>>>>> classes will be part of the platform team's repo and
> > > not
> > > > > >> >> the
> > > > > >> >>>>>>>>>> customer's
> > > > > >> >>>>>>>>>>> repo
> > > > > >> >>>>>>>>>>> - The triggers won't be able to use any libs other
> > than
> > > > std
> > > > > >> >>>> ones,
> > > > > >> >>>>>>>>>> which
> > > > > >> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> > > > > >> >>>>>>>>>>> - As we are the owners of the core Airflow repo,
> > > > customers
> > > > > >> >>> have
> > > > > >> >>>>> to
> > > > > >> >>>>>>>>>> get
> > > > > >> >>>>>>>>>>> our approval to land any class in this path (unlike
> > the
> > > > > >> >> dags
> > > > > >> >>>> repo
> > > > > >> >>>>>>>>>> which
> > > > > >> >>>>>>>>>>> they own)
> > > > > >> >>>>>>>>>>> - When a customer's task defer, we would have an
> > > > allowlist
> > > > > >> >> on
> > > > > >> >>>> our
> > > > > >> >>>>>>>>>> side
> > > > > >> >>>>>>>>>>> to check if we should do the async polling or not
> > > > > >> >>>>>>>>>>> - If the Trigger class isn't part of our repo
> > > > (allowlist),
> > > > > >> >>> just
> > > > > >> >>>>>>>>>> fail the
> > > > > >> >>>>>>>>>>> task, as anyway we won't be having the code that
> they
> > > > used
> > > > > >> >> in
> > > > > >> >>>> the
> > > > > >> >>>>>>>>>>> trigger
> > > > > >> >>>>>>>>>>> class
> > > > > >> >>>>>>>>>>> - If any of these conditions aren't suitable for you
> > > (as
> > > > a
> > > > > >> >>>>>>>>>> customer),
> > > > > >> >>>>>>>>>>> feel free to use sync tasks only
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> > > > > >> >> communicate
> > > > > >> >>>> over
> > > > > >> >>>>>>>>>> apis
> > > > > >> >>>>>>>>>>> only. If that is done, then we can have instances of
> > > > > >> >> triggerer
> > > > > >> >>>> svc
> > > > > >> >>>>>>>>>> running
> > > > > >> >>>>>>>>>>> at customer's side as well, which can process any
> type
> > > of
> > > > > >> >>> trigger
> > > > > >> >>>>>>>>>> class.
> > > > > >> >>>>>>>>>>> Though that's not a blocker for us at the moment,
> > cause
> > > > > >> >>> triggerer
> > > > > >> >>>>> are
> > > > > >> >>>>>>>>>>> mostly doing just polling using simple libs like
> > > requests.
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > > > >> >>>>>>>>>> <ikholopov@google.com.inva <mailto:
> ikholopov@google.com.inva>lid
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>> wrote:
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I
> > > believe
> > > > > it
> > > > > >> >>>>> aligns
> > > > > >> >>>>>>>>>> well
> > > > > >> >>>>>>>>>>>> with the goals of making Airflow well-scalable
> > beyond a
> > > > > >> >>>>> single-team
> > > > > >> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full
> > > > support
> > > > > >> >>> with
> > > > > >> >>>>> this
> > > > > >> >>>>>>>>>> one.
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>> I've left a couple of clarification requests on the
> > AIP
> > > > > >> >> page.
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>> Thanks,
> > > > > >> >>>>>>>>>>>> Igor
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > > > >> >>>>>>>>>>> sumeet.manit@gmail.com <mailto:
> sumeet.manit@gmail.com>>
> > > > > >> >>>>>>>>>>>> wrote:
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's
> > > good
> > > > to
> > > > > >> >>> know
> > > > > >> >>>>>>>>>> that
> > > > > >> >>>>>>>>>>> the
> > > > > >> >>>>>>>>>>>>> DAG processor has some preemptive measures in
> place
> > to
> > > > > >> >>> prevent
> > > > > >> >>>>>>>>>> access
> > > > > >> >>>>>>>>>>>>> to the DB. However, the main issue we are trying
> to
> > > > solve
> > > > > >> >> is
> > > > > >> >>>> not
> > > > > >> >>>>> to
> > > > > >> >>>>>>>>>>>> provide
> > > > > >> >>>>>>>>>>>>> DB creds to the customer teams, who are using
> > Airflow
> > > > as a
> > > > > >> >>>>>>>>>> multi-tenant
> > > > > >> >>>>>>>>>>>>> orchestration platform. I've updated the doc to
> > > reflect
> > > > > >> >> this
> > > > > >> >>>>> point
> > > > > >> >>>>>>>>>> as
> > > > > >> >>>>>>>>>>>> well.
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> Answering Jarek's points,
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token
> mechanism,
> > > > added
> > > > > >> >>> that
> > > > > >> >>>>> in
> > > > > >> >>>>>>>>>> doc,
> > > > > >> >>>>>>>>>>>> but
> > > > > >> >>>>>>>>>>>>> still how the token can be obtained (safely) is
> > still
> > > > open
> > > > > >> >> in
> > > > > >> >>>> my
> > > > > >> >>>>>>>>>> mind.
> > > > > >> >>>>>>>>>>> I
> > > > > >> >>>>>>>>>>>>> believe the token used by task executors can be
> > > created
> > > > > >> >>> outside
> > > > > >> >>>>> of
> > > > > >> >>>>>>>>>> it
> > > > > >> >>>>>>>>>>> as
> > > > > >> >>>>>>>>>>>>> well (I may be wrong here).
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> > > triggerers
> > > > > as
> > > > > >> >>>> well,
> > > > > >> >>>>>>>>>> but
> > > > > >> >>>>>>>>>>>> that
> > > > > >> >>>>>>>>>>>>> can be done as part of a different AIP
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work
> > largely.
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of
> dag_dirs
> > we
> > > > can
> > > > > >> >>> work
> > > > > >> >>>>>>>>>> with
> > > > > >> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would
> > be
> > > > > >> >> treated
> > > > > >> >>>> as
> > > > > >> >>>>> a
> > > > > >> >>>>>>>>>> diff
> > > > > >> >>>>>>>>>>>>> bundle.
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> Also, added points around callbacks, as these are
> > also
> > > > > >> >>> fetched
> > > > > >> >>>>>>>>>> directly
> > > > > >> >>>>>>>>>>>>> from the DB.
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > > > > >> >>>> jarek@potiuk.com <ma...@potiuk.com>>
> > > > > >> >>>>>>>>>>> wrote:
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today
> is
> > > > likely
> > > > > >> >>> not
> > > > > >> >>>>>>>>>>>> protection
> > > > > >> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> > does
> > > > > >> >>> protect
> > > > > >> >>>>>>>>>> against
> > > > > >> >>>>>>>>>>>>>> casual DB access attempts - the db session is
> > blanked
> > > > out
> > > > > >> >> in
> > > > > >> >>>> the
> > > > > >> >>>>>>>>>>>> parsing
> > > > > >> >>>>>>>>>>>>>> process , as are the env var configs
> > > > > >> >>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>
> > > > > >> >>>>>
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> <
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >
> > > > > >> >>>>>>>>>>>>>> -
> > > > > >> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no
> > > > protection
> > > > > >> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed
> > > back
> > > > > >> >> then
> > > > > >> >>> in
> > > > > >> >>>>>>>>>> March
> > > > > >> >>>>>>>>>>> I
> > > > > >> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with
> > > full
> > > > > >> >>>> knowledge
> > > > > >> >>>>>>>>>> it's
> > > > > >> >>>>>>>>>>>> not
> > > > > >> >>>>>>>>>>>>>> protecting against all threats.
> > > > > >> >>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash
> Berlin-Taylor <
> > > > > >> >>>>>>>>>> ash@apache.org <ma...@apache.org>>
> > > > > >> >>>>>>>>>>>>> wrote:
> > > > > >> >>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today
> is
> > > > likely
> > > > > >> >>> not
> > > > > >> >>>>>>>>>>>> protection
> > > > > >> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> > > does
> > > > > >> >>> protect
> > > > > >> >>>>>>>>>>> against
> > > > > >> >>>>>>>>>>>>>>> casual DB access attempts - the db session is
> > > blanked
> > > > > out
> > > > > >> >>> in
> > > > > >> >>>>>>>>>> the
> > > > > >> >>>>>>>>>>>>> parsing
> > > > > >> >>>>>>>>>>>>>>> process , as are the env var configs
> > > > > >> >>>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>
> > > > > >> >>>>>
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> <
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >
> > > > > >> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> > > > > >> >> protection
> > > > > >> >>>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> > > > > >> >> jarek@potiuk.com <ma...@potiuk.com>>
> > > > > >> >>>>>>>>>> wrote:
> > > > > >> >>>>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no
> > > > > built-in
> > > > > >> >>>>>>>>>>> protection
> > > > > >> >>>>>>>>>>>>>>> against
> > > > > >> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example -
> > read
> > > > > >> >>> database
> > > > > >> >>>>>>>>>>>>>>>> credentials from airflow configuration and use
> > them
> > > > to
> > > > > >> >>> talk
> > > > > >> >>>>>>>>>> to DB
> > > > > >> >>>>>>>>>>>>>>> directly.
> > > > > >> >>>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>>
> > > > > >> >>>>>>>>>>>
> > > > > >> >>>>>>>>>>
> > > > > >> >>>>>>>>>
> > > > > >> >>>>>>
> > > > > >> >>>>>>
> > > > > >> >>>>>
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by "Kataria, Ramit" <ra...@amazon.com.INVALID>.
I've been working on making Deadline callbacks work on the executor in addition to the Triggerer. That means introducing a new Workload type for the executor. I didn't want to tie it specifically to Deadlines and realized it would be a good idea to introduce generic Callbacks that the new workload can reference.
Here's an initial PR for adding a new Callback table and refactoring existing Deadline callbacks that run on the Triggerer to use these instead: https://github.com/apache/airflow/pull/54796
Ideally, once AIP-92 is in progress, these callbacks can be used or all the `on_*_callback`s as well. I want to ensure that these are generic enough for additional subclasses that would implement them. So, I'm looking for feedback/comments to align on a common definition for the new Workload type and the Callback model definition.
Thanks,
Ramit
On 2025-08-13, 1:39 AM, "Jarek Potiuk" <jarek@potiuk.com <ma...@potiuk.com>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque.
Those are all questions that we will open up for discussion once we get the
basic foundation :)
On Wed, Aug 13, 2025 at 10:16 AM Sumit Maheshwari <sumeet.manit@gmail.com <ma...@gmail.com>>
wrote:
> Awesome, having a standard approach for all kinds of authentication would
> be great, looking forward to it.
>
> BTW, on a side note, I see that as of now, things like Connections,
> Variables, and XComs which are present under Execution API namespace, don't
> have any authorization model (left with TODOs), so is there any plan how
> they will work, cause we might need something similar for
> Dag-processor/Triggerer as well.
>
> Also, as we've decided to create a diff API namespace and not use execution
> namespace for hosting new APIs required by dag-processor/triggerer, do
> we've to copy these classes & routes OR atleast refactor them, so they can
> serve all API namespaces.
>
> On Wed, Aug 13, 2025 at 1:27 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
>
> > Will do. We are also discussing - for now - within the security team -
> the
> > various aspects of authentication approach we want to have - for both
> > "UI/User authentication" as well as "Long running services" and how they
> > relate to token exchange, invalidation and other scenarios. What we will
> > come up - I hope shortly - is a proposal of general "model" of all kinds
> of
> > security and authentication scenarios, so that we do not have to reinvent
> > the wheel and try to figure out all the aspects with individual AIPs. We
> > are not far from bringing it to the open discussion at devlist, but we
> have
> > to be careful about some of the aspects that we might need to improve in
> > the current setup to close some - small - loopholes so bear with us :)
> >
> > On Wed, Aug 13, 2025 at 9:33 AM Sumit Maheshwari <sumeet.manit@gmail.com <ma...@gmail.com>
> >
> > wrote:
> >
> > > Thanks Ash and Jarek, for the detailed comments. I largely agree with
> the
> > > points mentioned by you guys, hence I updated the AIP and added a
> section
> > > on Authentication, API Versioning, and Packaging as well. Please go
> > through
> > > it once more and let me know if there are more things to consider
> before
> > I
> > > open it for voting.
> > >
> > >
> > > On Thu, Aug 7, 2025 at 6:03 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
> > >
> > > > Also
> > > >
> > > > > 1. The Authentication token. How will this long lived token work
> > > without
> > > > being insecure. Who and what will generate it? How will we identify
> > > > top-level requests for Variables in order to be able to add Variable
> > > > RBAC/ACLs. This is an important enough thing that I think it needs
> > > > discussion before we vote on this AIP.
> > > >
> > > > We are currently discussing - in the security team - approach for JWT
> > > token
> > > > handling, so likely we could move the discussion there, it does have
> > some
> > > > security implications and I think we should bring our finding to the
> > > > devlist when we complete it, but I think we should add this case
> there.
> > > > IMHO we should have a different approach for UI, different for Tasks,
> > > > different for Triggerer, and different for DagProcessor. (possibly
> the
> > > > Trigerer and DagProcessor could be the same because they share
> > > essentially
> > > > the same long-living token. Ash - I will add this to the discussion
> > > there.
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > > On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>
> wrote:
> > > >
> > > > > Ah.. So if we are talking about a more complete approach - seeing
> > those
> > > > > comments from Ash - make me think if we should have another AIP.
> > > > > (connected) about splitting the distributions. We have never
> > finalized
> > > it
> > > > > (nor even discussed id) but Ash - you had some initial document for
> > > that.
> > > > > So maybe we should finalize it and rather than specify it in this
> > AIP -
> > > > > have a separate AIP about distribution split that AIP-92 could
> depend
> > > on.
> > > > > It seems much more reasonable to split "distribution and code
> split"
> > > from
> > > > > parsing isolation I think and implement them separately/in
> parallel.
> > > > >
> > > > > Reading Ash comments (and maybe I am going a bit further than Ash)
> it
> > > > > calls for something that I am a big proponent of - splitting
> > > > "airflow-core"
> > > > > and having a different "scheduler". "webserver", "dag processor"
> and
> > > > > "triggerer" distributions. Now - we have the capability of having
> > > > "shared"
> > > > > code - we do not need "common" code to make it happen - because we
> > can
> > > > > share code.
> > > > >
> > > > > What it could give us - on top of clean client/server split, we
> could
> > > > have
> > > > > different dependencies used by those distributions. Additionally,
> we
> > > > could
> > > > > also split-off the executors from providers and finally implement
> it
> > in
> > > > the
> > > > > way that scheduler does not use providers at all (not even
> > > > cncf.kubernetes
> > > > > nor celery providers installed in scheduler nor webserver but
> > > "executors"
> > > > > packages instead. The code sharing approach with symlinks we have
> now
> > > > will
> > > > > make it a .... breeze :) . That would also imply sharing
> "connection"
> > > > > definitions through DB, and likely implementing "test connection"
> > > > > feature properly finally (i.e executing test connection in worker /
> > > > > triggerer rather than in web server which is a reason why we
> disabled
> > > it
> > > > by
> > > > > default now). This way "api-server" would not need any of the
> > providers
> > > > to
> > > > > be installed either which IMHO is the biggest win from a security
> > point
> > > > of
> > > > > view.
> > > > >
> > > > > And the nice thing about it is that it would be rather transparent
> > when
> > > > > anyone uses "pip install apache-airflow" - it would behave exactly
> > the
> > > > > same, no more complexity involved, simply more distributions
> > installed
> > > > when
> > > > > 'apache-airflow" meta-distribution is used, but it would allow
> those
> > > who
> > > > > want to implement a more complex and secure setup to have different
> > > > > "environments" with modularized pieces of airflow installed - only
> > > > > "apache-airflow-dag-processor + task-sdk + providers" where
> > > dag-processor
> > > > > is run, only "apache-airflow-scheduler + executors" where scheduler
> > is
> > > > > installed only "apache-airflow-task-sdk + providers" where workers
> > are
> > > > > running, only "apache-airflow-api-server" where api-server is
> running
> > > and
> > > > > only "apache-airflow-trigger + task-sdk + providers" .
> > > > >
> > > > > I am happy (Ash If you are fine with that) to take that original
> > > document
> > > > > over and lead this part and new AIP to completion (including
> > > > > implementation), I am very much convinced that this will lead to
> much
> > > > > better dependency security and more modular code without impacting
> > the
> > > > > "apache-airflow" installation complexity.
> > > > >
> > > > > If we do it this way- the part of code/clean split would be
> > "delegated
> > > > > out" from AIP-92 to this new AIP and turned into dependency.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <ash@apache.org <ma...@apache.org>>
> > > wrote:
> > > > >
> > > > >> This AIP is definitely heading in the right direction and is a
> > feature
> > > > >> I’d like to see.
> > > > >>
> > > > >> For me the outstanding things that need more detail:
> > > > >>
> > > > >> 1. The Authentication token. How will this long lived token work
> > > without
> > > > >> being insecure. Who and what will generate it? How will we
> identify
> > > > >> top-level requests for Variables in order to be able to add
> Variable
> > > > >> RBAC/ACLs. This is an important enough thing that I think it needs
> > > > >> discussion before we vote on this AIP.
> > > > >> 2. Security generally — how will this work, especially with the
> > > > >> multi-team? I think this likely means making the APIs work on the
> > > bundle
> > > > >> level as you mention in the doc, but I haven’t thought deeply
> about
> > > this
> > > > >> yet.
> > > > >> 3. API Versioning? One of the the key driving goals with AIP-72
> and
> > > the
> > > > >> Task Execution SDK was the idea that “you can upgrade the API
> server
> > > as
> > > > you
> > > > >> like, and your clients/workers never need to work” — i.e. the API
> > > > server is
> > > > >> 100% working with all older versions of the TaskSDK. I don’t know
> if
> > > we
> > > > >> will achieve that goal in the long run but it is the desire, and
> > part
> > > of
> > > > >> why we are using CalVer and the Cadwyn library to provide API
> > > > versioning.
> > > > >> 4. As mentioned previously, not sure the existing serialised JSON
> > > format
> > > > >> for DAGs is correct, but since that now has version and we already
> > > have
> > > > the
> > > > >> ability to upgrade that somewhere in the Airflow Core that doesn’t
> > > > >> necessarily become a blocker/pre-requisite for this AIP.
> > > > >>
> > > > >> I think Dag parsing API client+submission+parsing process manager
> > > should
> > > > >> either live in the Task SDK dist, or in a new separate dist that
> > uses
> > > > >> TaskSDK, but crucially not in apache-airflow-core. My reason for
> > this
> > > is
> > > > >> that I want it to be possible for the server components
> (scheduler,
> > > API
> > > > >> server) to not need task-sdk installed (just for
> > cleanliness/avoiding
> > > > >> confusion about what versions it needs) and also vice-verse, to be
> > > able
> > > > to
> > > > >> run a “team worker bundle” (Dag parsing, workers, triggered/async
> > > > workers)
> > > > >> on whatever version of TaskSDK they choose, again without
> > > > >> apache-airflow-core installed for avoidance of doubt.
> > > > >>
> > > > >> Generally I would like this as it means we can have a nicer
> > separation
> > > > of
> > > > >> Core and Dag parsing code, as the dag parsing itself uses the SDK,
> > it
> > > > would
> > > > >> be nice to have a proper server/client split, both from a tighter
> > > > security
> > > > >> point-of-view, but also from a code layout point of view.
> > > > >>
> > > > >> -ash
> > > > >>
> > > > >>
> > > > >> > On 7 Aug 2025, at 12:36, Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>> wrote:
> > > > >> >
> > > > >> > Well, you started it - so it's up to you to decide if you think
> we
> > > > have
> > > > >> > consensus, or whether we need a vote.
> > > > >> >
> > > > >> > And It's not a question of "informal" vote but it's rather clear
> > > > >> following
> > > > >> > the https://www.apache.org/foundation/voting.html <https://www.apache.org/foundation/voting.html> that we
> either
> > > > need a
> > > > >> > LAZY CONSENSUS or VOTE thread. Both are formal.
> > > > >> >
> > > > >> > This is the difficult part when you have a proposal, to assess
> (by
> > > > you)
> > > > >> > whether we are converging to consensus or whether vote is
> needed.
> > > > There
> > > > >> is
> > > > >> > no other body or "authority" to do it for you.
> > > > >> >
> > > > >> > J.
> > > > >> >
> > > > >> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <
> > > > sumeet.manit@gmail.com <ma...@gmail.com>
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Sorry for nudging again, but can we get into some consensus on
> > > this?
> > > > I
> > > > >> mean
> > > > >> >> if this AIP isn't good enough, then we can drop it altogether
> and
> > > > >> someone
> > > > >> >> can rethink the whole thing. Should we do some kind of informal
> > > > voting
> > > > >> and
> > > > >> >> close this thread?
> > > > >> >>
> > > > >> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <jarek@potiuk.com <ma...@potiuk.com>>
> > > > wrote:
> > > > >> >>
> > > > >> >>>>> My main concern with this right now is the serialisation
> > format
> > > of
> > > > >> >> DAGs
> > > > >> >>> —
> > > > >> >>>> it wasn’t really designed with remote submission in mind, so
> it
> > > > need
> > > > >> >> some
> > > > >> >>>> careful examination to see if it is fit for this purpose or
> > not.
> > > > >> >>>
> > > > >> >>> I understand Ash's concerns - the format has not been designed
> > > with
> > > > >> >>> size/speed optimization in mind so **possibly** we could
> design
> > a
> > > > >> >> different
> > > > >> >>> format that would be better suited.
> > > > >> >>>
> > > > >> >>> BUT ... Done is better than perfect.
> > > > >> >>>
> > > > >> >>> I think there are a number of risks involved in changing the
> > > format
> > > > >> and
> > > > >> >> it
> > > > >> >>> could significantly increase time of development with
> uncertain
> > > > gains
> > > > >> at
> > > > >> >>> the end - also because of the progress in compression that
> > > happened
> > > > >> over
> > > > >> >>> the last few years.
> > > > >> >>>
> > > > >> >>> It might be a good idea to experiment a bit with different
> > > > compression
> > > > >> >>> algorithms for "our" dag representation and possibly we could
> > find
> > > > the
> > > > >> >> best
> > > > >> >>> algorithm for "airflow dag" type of json data. There are a lot
> > of
> > > > >> >>> repetitions in the JSON representation and I guess in "our"
> json
> > > > >> >>> representation there are some artifacts and repeated sections
> > that
> > > > >> simply
> > > > >> >>> might compress well with different algorithms. Also in this
> case
> > > > >> >>> speed matters (and CPU trade-off).
> > > > >> >>>
> > > > >> >>> Looking at compression "theory" - before we experiment with
> it -
> > > > >> there is
> > > > >> >>> the relatively new standard "zstandard"
> > > > >> https://github.com/facebook/zstd <https://github.com/facebook/zstd>
> > > > >> >>> compression opensourced in 2016 which I've heard good things
> > > about -
> > > > >> >>> especially that it maintains a very good compression rate for
> > text
> > > > >> data,
> > > > >> >>> but also it is tremendously fast - especially for
> decompression
> > > > >> (which is
> > > > >> >>> super important factor for us - we compress new DAG
> > representation
> > > > far
> > > > >> >> less
> > > > >> >>> often than decompress it in general case). It is standardized
> in
> > > RFC
> > > > >> >>> https://datatracker.ietf.org/doc/html/rfc8878 <https://datatracker.ietf.org/doc/html/rfc8878> and there are
> > > various
> > > > >> >>> implementations and it is even being added to Python standard
> > > > library
> > > > >> in
> > > > >> >>> Python 3.14
> > > > >> https://docs.python.org/3.14/library/compression.zstd.html <https://docs.python.org/3.14/library/compression.zstd.html>
> > > > >> >> and
> > > > >> >>> there is a very well maintained python binding library
> > > > >> >>> https://pypi.org/project/zstd/ <https://pypi.org/project/zstd/> to Yann Collet (algorithm
> > author)
> > > > >> ZSTD C
> > > > >> >>> library. And libzstd is already part of our images - it is
> > needed
> > > by
> > > > >> >> other
> > > > >> >>> dependencies of ours. All with BSD licence, directly usable by
> > us.
> > > > >> >>>
> > > > >> >>> I think this one might be a good candidate for us to try, and
> > > > possibly
> > > > >> >> with
> > > > >> >>> zstd we could achieve both size and CPU overhead that would be
> > > > >> comparable
> > > > >> >>> with any "new" format we could come up with - especially that
> we
> > > are
> > > > >> >>> talking merely about processing a huge blob between "storable"
> > > > >> >> (compressed)
> > > > >> >>> and "locally usable" state (Python dict). We could likely use
> a
> > > > >> streaming
> > > > >> >>> JSON library (say the one that is used in Pydantic internally
> > > > >> >>> https://github.com/pydantic/jiter <https://github.com/pydantic/jiter> - we already have it as
> part
> > of
> > > > >> >>> Pydantic)
> > > > >> >>> to also save memory - we could stream decompressed stream into
> > > > jitter
> > > > >> so
> > > > >> >>> that both the json dict and string representation does not
> have
> > to
> > > > be
> > > > >> >>> loaded fully in memory at the same time. There are likely lots
> > of
> > > > >> >>> optimisations we could do - I mentioned possibly streaming the
> > > data
> > > > >> from
> > > > >> >>> API directly to DB (if this is possible - not sure)
> > > > >> >>>
> > > > >> >>> J.
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> > > > >> sumeet.manit@gmail.com <ma...@gmail.com>>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >> >>>>>
> > > > >> >>>>> My main concern with this right now is the serialisation
> > format
> > > of
> > > > >> >>> DAGs —
> > > > >> >>>>> it wasn’t really designed with remote submission in mind, so
> > it
> > > > need
> > > > >> >>> some
> > > > >> >>>>> careful examination to see if it is fit for this purpose or
> > not.
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>> I'm not sure on this point, cause if we are able to convert a
> > DAG
> > > > >> into
> > > > >> >>>> JSON, then it has to be transferable over the internet.
> > > > >> >>>>
> > > > >> >>>> In particular One of the things I worry about is that the
> JSON
> > > can
> > > > >> get
> > > > >> >>> huge
> > > > >> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> Yeah, agree on this, thats why we can transfer compressed
> data
> > > > >> instead
> > > > >> >> of
> > > > >> >>>> real json. Of course, this won't guarantee that the payload
> > will
> > > > >> always
> > > > >> >>> be
> > > > >> >>>> small enough, but we can't say that it'll definitely happen
> > > either.
> > > > >> >>>>
> > > > >> >>>> I also wonder if as part of this proposal we should move the
> > > > Callback
> > > > >> >>>>> requests off the dag parsers and on to the workers instead
> > > > >> >>>>
> > > > >> >>>> let's make such a "workfload" implementation stream that
> could
> > > > >> support
> > > > >> >>> both
> > > > >> >>>>> - Deadlines and DAG parsing logic
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> I don't have any strong opinion here, but it feels like it's
> > > gonna
> > > > >> blow
> > > > >> >>> up
> > > > >> >>>> the scope of the AIP too much.
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <
> jarek@potiuk.com <ma...@potiuk.com>>
> > > > >> wrote:
> > > > >> >>>>
> > > > >> >>>>>> My main concern with this right now is the serialisation
> > format
> > > > of
> > > > >> >>>> DAGs —
> > > > >> >>>>> it wasn’t really designed with remote submission in mind, so
> > it
> > > > need
> > > > >> >>> some
> > > > >> >>>>> careful examination to see if it is fit for this purpose or
> > not.
> > > > >> >>>>>
> > > > >> >>>>> Yep. That might be potentially a problem (or at least "need
> > more
> > > > >> >>>> resources
> > > > >> >>>>> to run airflow") and that is where my "2x memory" came from
> if
> > > we
> > > > do
> > > > >> >> it
> > > > >> >>>> in
> > > > >> >>>>> a trivial way. Currently we a) keep the whole DAG in memory
> > when
> > > > >> >>>>> serializing it b) submit it to database (also using
> > essentially
> > > > some
> > > > >> >>> kind
> > > > >> >>>>> of API (implemented by the database client) - so we know the
> > > whole
> > > > >> >>> thing
> > > > >> >>>>> "might work" but indeed if you use a trivial implementation
> of
> > > > >> >>> submitting
> > > > >> >>>>> the whole json - it basically means that the whole json will
> > > have
> > > > to
> > > > >> >>> also
> > > > >> >>>>> be kept in the memory of API server. But we also compress it
> > > when
> > > > >> >>> needed
> > > > >> >>>> -
> > > > >> >>>>> I wonder what are the compression ratios we saw with those
> > > > 10-20MBs
> > > > >> >>> Dags
> > > > >> >>>> -
> > > > >> >>>>> if the problem is using strings where bool would suffice,
> > > > >> compression
> > > > >> >>>>> should generally help a lot. We could only ever send
> > compressed
> > > > data
> > > > >> >>> over
> > > > >> >>>>> the API - there seems to be no need to send "plain JSON"
> data
> > > over
> > > > >> >> the
> > > > >> >>>> API
> > > > >> >>>>> or storing the plain JSON in the DB (of course that trades
> > > memory
> > > > >> for
> > > > >> >>>> CPU).
> > > > >> >>>>>
> > > > >> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres)
> have
> > > > >> >> support
> > > > >> >>>> for
> > > > >> >>>>> any kind if binary data streaming - because that could help
> a
> > > lot
> > > > of
> > > > >> >> if
> > > > >> >>>> we
> > > > >> >>>>> could use streaming HTTP API and chunk and append the binary
> > > > chunks
> > > > >> >>> (when
> > > > >> >>>>> writing) - or read data in chunks ans stream them back via
> the
> > > > API.
> > > > >> >>> That
> > > > >> >>>>> could seriously decrease the amount of memory needed by the
> > API
> > > > >> >> server
> > > > >> >>> to
> > > > >> >>>>> process such huge serialized dags.
> > > > >> >>>>>
> > > > >> >>>>> And yeah - I would also love the "execute task" to be
> > > implemented
> > > > >> >> here
> > > > >> >>> -
> > > > >> >>>>> but I am not sure if this should be part of the same effort
> or
> > > > maybe
> > > > >> >> a
> > > > >> >>>>> separate implementation? That sounds very loosely coupled
> with
> > > DB
> > > > >> >>>>> isolation. And it seems a common theme - I think that would
> > also
> > > > >> make
> > > > >> >>> the
> > > > >> >>>>> sync Deadline alerts case that we discussed at the dev call
> > > > today. I
> > > > >> >>>> wonder
> > > > >> >>>>> if that should not be kind of parallel (let's make such a
> > > > >> "workfload"
> > > > >> >>>>> implementation stream that could support both - Deadlines
> and
> > > DAG
> > > > >> >>> parsing
> > > > >> >>>>> logic. We have already two "users" for it and I really love
> > the
> > > > >> >> saying
> > > > >> >>>> "if
> > > > >> >>>>> you want to make something reusable - make it usable
> first" -
> > > > seems
> > > > >> >>> like
> > > > >> >>>>> we might have good opportunity to make such workload
> > > > implementation
> > > > >> >>>> "doubly
> > > > >> >>>>> used" from the beginning which would increase chances it
> will
> > > be
> > > > >> >>>>> "reusable" for other things as well :).
> > > > >> >>>>>
> > > > >> >>>>> J.
> > > > >> >>>>>
> > > > >> >>>>>
> > > > >> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <
> > > > ash@apache.org <ma...@apache.org>>
> > > > >> >>>> wrote:
> > > > >> >>>>>
> > > > >> >>>>>> My main concern with this right now is the serialisation
> > format
> > > > of
> > > > >> >>>> DAGs —
> > > > >> >>>>>> it wasn’t really designed with remote submission in mind,
> so
> > it
> > > > >> >> need
> > > > >> >>>> some
> > > > >> >>>>>> careful examination to see if it is fit for this purpose or
> > > not.
> > > > >> >>>>>>
> > > > >> >>>>>> In particular One of the things I worry about is that the
> > JSON
> > > > can
> > > > >> >>> get
> > > > >> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!)
> > > > (which
> > > > >> >> is
> > > > >> >>>>>> likely due to things being included as text when a bool
> might
> > > > >> >>> suffice,
> > > > >> >>>>> for
> > > > >> >>>>>> example) But I don’t think “just submit the existing JSON
> > over
> > > an
> > > > >> >>> API”
> > > > >> >>>>> is a
> > > > >> >>>>>> good idea.
> > > > >> >>>>>>
> > > > >> >>>>>> I also wonder if as part of this proposal we should move
> the
> > > > >> >> Callback
> > > > >> >>>>>> requests off the dag parsers and on to the workers instead
> —
> > in
> > > > >> >>> AIP-72
> > > > >> >>>> we
> > > > >> >>>>>> introduced the concept of a Workload, with the only one
> > > existing
> > > > >> >>> right
> > > > >> >>>>> now
> > > > >> >>>>>> is “ExecuteTask”
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88 <https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88>
> > > > >> >>>>>> — it might be time to finally move task and dag callbacks
> to
> > > the
> > > > >> >> same
> > > > >> >>>>> thing
> > > > >> >>>>>> and make dag parsers only responsible for, well, parsing.
> :)
> > > > >> >>>>>>
> > > > >> >>>>>> These are all solvable problems, and this will be a great
> > > feature
> > > > >> >> to
> > > > >> >>>>> have,
> > > > >> >>>>>> but we need to do some more thinking and planning first.
> > > > >> >>>>>>
> > > > >> >>>>>> -ash
> > > > >> >>>>>>
> > > > >> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> > > > >> >> sumeet.manit@gmail.com <ma...@gmail.com>
> > > > >> >>>>
> > > > >> >>>>>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>> Gentle reminder for everyone to review the proposal.
> > > > >> >>>>>>>
> > > > >> >>>>>>> Updated link:
> > > > >> >>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services <https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services>
> > > > >> >>>>>>>
> > > > >> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > > > >> >>>>> sumeet.manit@gmail.com <ma...@gmail.com>
> > > > >> >>>>>>>
> > > > >> >>>>>>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and
> others
> > > > >> >>>>> suggested, I
> > > > >> >>>>>>>> expanded the scope of this AIP and divided it into three
> > > > phases.
> > > > >> >>>> With
> > > > >> >>>>>> the
> > > > >> >>>>>>>> increased scope, the boundary line between this AIP and
> > > AIP-85
> > > > >> >>> got a
> > > > >> >>>>>> little
> > > > >> >>>>>>>> thinner, but I believe these are still two different
> > > > >> >> enhancements
> > > > >> >>> to
> > > > >> >>>>>> make.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > > >> >>>>>> sumeet.manit@gmail.com <ma...@gmail.com>>
> > > > >> >>>>>>>> wrote:
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well
> > to
> > > be
> > > > >> >>> part
> > > > >> >>>>> of
> > > > >> >>>>>>>>> this AIP and phase out the implementation. Though I
> didn't
> > > > >> >>> exclude
> > > > >> >>>>>> Triggers
> > > > >> >>>>>>>>> because "Uber" doesn't need that, I just thought of
> > keeping
> > > > the
> > > > >> >>>> scope
> > > > >> >>>>>> of
> > > > >> >>>>>>>>> development small and achieving them, just like it was
> > done
> > > in
> > > > >> >>>>> Airlfow
> > > > >> >>>>>> 3 by
> > > > >> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> But if you think Triggers should be part of this AIP
> > itself,
> > > > >> >>> then I
> > > > >> >>>>> can
> > > > >> >>>>>>>>> do that and include Triggers as well in it.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <
> > > > jarek@potiuk.com <ma...@potiuk.com>
> > > > >> >>>
> > > > >> >>>>> wrote:
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>> I would very much prefer the architectural choices of
> > this
> > > > AIP
> > > > >> >>> are
> > > > >> >>>>>> based
> > > > >> >>>>>>>>>> on
> > > > >> >>>>>>>>>> "general public" needs rather than "Uber needs" even if
> > > Uber
> > > > >> >>> will
> > > > >> >>>> be
> > > > >> >>>>>>>>>> implementing it - so from my point of view having
> Trigger
> > > > >> >>>> separation
> > > > >> >>>>>> as
> > > > >> >>>>>>>>>> part of it is quite important.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> But that's not even this.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> We've been discussing for example for Deadlines (being
> > > > >> >>> implemented
> > > > >> >>>>> by
> > > > >> >>>>>>>>>> Dennis and Ramit a possibility of short,
> > > notification-style
> > > > >> >>>>>> "deadlines"
> > > > >> >>>>>>>>>> to be send to triggerer for execution - this is well
> > > advanced
> > > > >> >>> now,
> > > > >> >>>>> and
> > > > >> >>>>>>>>>> whether you want it or not Dag-provided code might be
> > > > >> >> serialized
> > > > >> >>>> and
> > > > >> >>>>>> sent
> > > > >> >>>>>>>>>> to triggerer for execution. This is part of our
> "broader"
> > > > >> >>>>>> architectural
> > > > >> >>>>>>>>>> change where we treat "workers" and "triggerer"
> similarly
> > > as
> > > > a
> > > > >> >>>>> general
> > > > >> >>>>>>>>>> executors of "sync" and "async" tasks respectively.
> > That's
> > > > >> >> where
> > > > >> >>>>>> Airflow
> > > > >> >>>>>>>>>> is
> > > > >> >>>>>>>>>> evolving towards - inevitably.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> But we can of course phase things in out for
> > > implementation -
> > > > >> >>> even
> > > > >> >>>>> if
> > > > >> >>>>>> AIP
> > > > >> >>>>>>>>>> should cover both, I think if the goal of the AIP and
> > > > preamble
> > > > >> >>> is
> > > > >> >>>>>> about
> > > > >> >>>>>>>>>> separating "user code" from "database" as the main
> > reason,
> > > it
> > > > >> >>> also
> > > > >> >>>>>> means
> > > > >> >>>>>>>>>> Triggerer if you ask me (from design point of view at
> > > least).
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Again implementation can be phased and even different
> > > people
> > > > >> >> and
> > > > >> >>>>> teams
> > > > >> >>>>>>>>>> might work on those phases/pieces.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> J.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > > >> >>>>>> sumeet.manit@gmail.com <ma...@gmail.com>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> > triggerers
> > > > as
> > > > >> >>>> well,
> > > > >> >>>>>>>>>> but
> > > > >> >>>>>>>>>>>> that
> > > > >> >>>>>>>>>>>> can be done as part of a different AIP
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> You won't achieve your goal of "true" isolation of
> user
> > > code
> > > > >> >> if
> > > > >> >>>> you
> > > > >> >>>>>>>>>> don't
> > > > >> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it -
> it
> > > > >> >> should
> > > > >> >>>>> cover
> > > > >> >>>>>>>>>>> both.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> My bad, should've explained our architecture for
> > triggers
> > > as
> > > > >> >>>> well,
> > > > >> >>>>>>>>>>> apologies. So here it is:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> - Triggers would be running on a centralized service,
> > so
> > > > >> >> all
> > > > >> >>>> the
> > > > >> >>>>>>>>>> Trigger
> > > > >> >>>>>>>>>>> classes will be part of the platform team's repo and
> > not
> > > > >> >> the
> > > > >> >>>>>>>>>> customer's
> > > > >> >>>>>>>>>>> repo
> > > > >> >>>>>>>>>>> - The triggers won't be able to use any libs other
> than
> > > std
> > > > >> >>>> ones,
> > > > >> >>>>>>>>>> which
> > > > >> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> > > > >> >>>>>>>>>>> - As we are the owners of the core Airflow repo,
> > > customers
> > > > >> >>> have
> > > > >> >>>>> to
> > > > >> >>>>>>>>>> get
> > > > >> >>>>>>>>>>> our approval to land any class in this path (unlike
> the
> > > > >> >> dags
> > > > >> >>>> repo
> > > > >> >>>>>>>>>> which
> > > > >> >>>>>>>>>>> they own)
> > > > >> >>>>>>>>>>> - When a customer's task defer, we would have an
> > > allowlist
> > > > >> >> on
> > > > >> >>>> our
> > > > >> >>>>>>>>>> side
> > > > >> >>>>>>>>>>> to check if we should do the async polling or not
> > > > >> >>>>>>>>>>> - If the Trigger class isn't part of our repo
> > > (allowlist),
> > > > >> >>> just
> > > > >> >>>>>>>>>> fail the
> > > > >> >>>>>>>>>>> task, as anyway we won't be having the code that they
> > > used
> > > > >> >> in
> > > > >> >>>> the
> > > > >> >>>>>>>>>>> trigger
> > > > >> >>>>>>>>>>> class
> > > > >> >>>>>>>>>>> - If any of these conditions aren't suitable for you
> > (as
> > > a
> > > > >> >>>>>>>>>> customer),
> > > > >> >>>>>>>>>>> feel free to use sync tasks only
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> > > > >> >> communicate
> > > > >> >>>> over
> > > > >> >>>>>>>>>> apis
> > > > >> >>>>>>>>>>> only. If that is done, then we can have instances of
> > > > >> >> triggerer
> > > > >> >>>> svc
> > > > >> >>>>>>>>>> running
> > > > >> >>>>>>>>>>> at customer's side as well, which can process any type
> > of
> > > > >> >>> trigger
> > > > >> >>>>>>>>>> class.
> > > > >> >>>>>>>>>>> Though that's not a blocker for us at the moment,
> cause
> > > > >> >>> triggerer
> > > > >> >>>>> are
> > > > >> >>>>>>>>>>> mostly doing just polling using simple libs like
> > requests.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > > >> >>>>>>>>>> <ikholopov@google.com.inva <ma...@google.com.inva>lid
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I
> > believe
> > > > it
> > > > >> >>>>> aligns
> > > > >> >>>>>>>>>> well
> > > > >> >>>>>>>>>>>> with the goals of making Airflow well-scalable
> beyond a
> > > > >> >>>>> single-team
> > > > >> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full
> > > support
> > > > >> >>> with
> > > > >> >>>>> this
> > > > >> >>>>>>>>>> one.
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> I've left a couple of clarification requests on the
> AIP
> > > > >> >> page.
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Thanks,
> > > > >> >>>>>>>>>>>> Igor
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > > >> >>>>>>>>>>> sumeet.manit@gmail.com <ma...@gmail.com>>
> > > > >> >>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's
> > good
> > > to
> > > > >> >>> know
> > > > >> >>>>>>>>>> that
> > > > >> >>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>> DAG processor has some preemptive measures in place
> to
> > > > >> >>> prevent
> > > > >> >>>>>>>>>> access
> > > > >> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to
> > > solve
> > > > >> >> is
> > > > >> >>>> not
> > > > >> >>>>> to
> > > > >> >>>>>>>>>>>> provide
> > > > >> >>>>>>>>>>>>> DB creds to the customer teams, who are using
> Airflow
> > > as a
> > > > >> >>>>>>>>>> multi-tenant
> > > > >> >>>>>>>>>>>>> orchestration platform. I've updated the doc to
> > reflect
> > > > >> >> this
> > > > >> >>>>> point
> > > > >> >>>>>>>>>> as
> > > > >> >>>>>>>>>>>> well.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Answering Jarek's points,
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism,
> > > added
> > > > >> >>> that
> > > > >> >>>>> in
> > > > >> >>>>>>>>>> doc,
> > > > >> >>>>>>>>>>>> but
> > > > >> >>>>>>>>>>>>> still how the token can be obtained (safely) is
> still
> > > open
> > > > >> >> in
> > > > >> >>>> my
> > > > >> >>>>>>>>>> mind.
> > > > >> >>>>>>>>>>> I
> > > > >> >>>>>>>>>>>>> believe the token used by task executors can be
> > created
> > > > >> >>> outside
> > > > >> >>>>> of
> > > > >> >>>>>>>>>> it
> > > > >> >>>>>>>>>>> as
> > > > >> >>>>>>>>>>>>> well (I may be wrong here).
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> > triggerers
> > > > as
> > > > >> >>>> well,
> > > > >> >>>>>>>>>> but
> > > > >> >>>>>>>>>>>> that
> > > > >> >>>>>>>>>>>>> can be done as part of a different AIP
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work
> largely.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs
> we
> > > can
> > > > >> >>> work
> > > > >> >>>>>>>>>> with
> > > > >> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would
> be
> > > > >> >> treated
> > > > >> >>>> as
> > > > >> >>>>> a
> > > > >> >>>>>>>>>> diff
> > > > >> >>>>>>>>>>>>> bundle.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Also, added points around callbacks, as these are
> also
> > > > >> >>> fetched
> > > > >> >>>>>>>>>> directly
> > > > >> >>>>>>>>>>>>> from the DB.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > > > >> >>>> jarek@potiuk.com <ma...@potiuk.com>>
> > > > >> >>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> > > likely
> > > > >> >>> not
> > > > >> >>>>>>>>>>>> protection
> > > > >> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> does
> > > > >> >>> protect
> > > > >> >>>>>>>>>> against
> > > > >> >>>>>>>>>>>>>> casual DB access attempts - the db session is
> blanked
> > > out
> > > > >> >> in
> > > > >> >>>> the
> > > > >> >>>>>>>>>>>> parsing
> > > > >> >>>>>>>>>>>>>> process , as are the env var configs
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316 <https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316>
> > > > >> >>>>>>>>>>>>>> -
> > > > >> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no
> > > protection
> > > > >> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed
> > back
> > > > >> >> then
> > > > >> >>> in
> > > > >> >>>>>>>>>> March
> > > > >> >>>>>>>>>>> I
> > > > >> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with
> > full
> > > > >> >>>> knowledge
> > > > >> >>>>>>>>>> it's
> > > > >> >>>>>>>>>>>> not
> > > > >> >>>>>>>>>>>>>> protecting against all threats.
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > > > >> >>>>>>>>>> ash@apache.org <ma...@apache.org>>
> > > > >> >>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> > > likely
> > > > >> >>> not
> > > > >> >>>>>>>>>>>> protection
> > > > >> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> > does
> > > > >> >>> protect
> > > > >> >>>>>>>>>>> against
> > > > >> >>>>>>>>>>>>>>> casual DB access attempts - the db session is
> > blanked
> > > > out
> > > > >> >>> in
> > > > >> >>>>>>>>>> the
> > > > >> >>>>>>>>>>>>> parsing
> > > > >> >>>>>>>>>>>>>>> process , as are the env var configs
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316 <https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316>
> > > > >> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> > > > >> >> protection
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> > > > >> >> jarek@potiuk.com <ma...@potiuk.com>>
> > > > >> >>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no
> > > > built-in
> > > > >> >>>>>>>>>>> protection
> > > > >> >>>>>>>>>>>>>>> against
> > > > >> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example -
> read
> > > > >> >>> database
> > > > >> >>>>>>>>>>>>>>>> credentials from airflow configuration and use
> them
> > > to
> > > > >> >>> talk
> > > > >> >>>>>>>>>> to DB
> > > > >> >>>>>>>>>>>>>>> directly.
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > > >>
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
Those are all questions that we will open up for discussion once we get the
basic foundation :)
On Wed, Aug 13, 2025 at 10:16 AM Sumit Maheshwari <su...@gmail.com>
wrote:
> Awesome, having a standard approach for all kinds of authentication would
> be great, looking forward to it.
>
> BTW, on a side note, I see that as of now, things like Connections,
> Variables, and XComs which are present under Execution API namespace, don't
> have any authorization model (left with TODOs), so is there any plan how
> they will work, cause we might need something similar for
> Dag-processor/Triggerer as well.
>
> Also, as we've decided to create a diff API namespace and not use execution
> namespace for hosting new APIs required by dag-processor/triggerer, do
> we've to copy these classes & routes OR atleast refactor them, so they can
> serve all API namespaces.
>
> On Wed, Aug 13, 2025 at 1:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Will do. We are also discussing - for now - within the security team -
> the
> > various aspects of authentication approach we want to have - for both
> > "UI/User authentication" as well as "Long running services" and how they
> > relate to token exchange, invalidation and other scenarios. What we will
> > come up - I hope shortly - is a proposal of general "model" of all kinds
> of
> > security and authentication scenarios, so that we do not have to reinvent
> > the wheel and try to figure out all the aspects with individual AIPs. We
> > are not far from bringing it to the open discussion at devlist, but we
> have
> > to be careful about some of the aspects that we might need to improve in
> > the current setup to close some - small - loopholes so bear with us :)
> >
> > On Wed, Aug 13, 2025 at 9:33 AM Sumit Maheshwari <sumeet.manit@gmail.com
> >
> > wrote:
> >
> > > Thanks Ash and Jarek, for the detailed comments. I largely agree with
> the
> > > points mentioned by you guys, hence I updated the AIP and added a
> section
> > > on Authentication, API Versioning, and Packaging as well. Please go
> > through
> > > it once more and let me know if there are more things to consider
> before
> > I
> > > open it for voting.
> > >
> > >
> > > On Thu, Aug 7, 2025 at 6:03 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Also
> > > >
> > > > > 1. The Authentication token. How will this long lived token work
> > > without
> > > > being insecure. Who and what will generate it? How will we identify
> > > > top-level requests for Variables in order to be able to add Variable
> > > > RBAC/ACLs. This is an important enough thing that I think it needs
> > > > discussion before we vote on this AIP.
> > > >
> > > > We are currently discussing - in the security team - approach for JWT
> > > token
> > > > handling, so likely we could move the discussion there, it does have
> > some
> > > > security implications and I think we should bring our finding to the
> > > > devlist when we complete it, but I think we should add this case
> there.
> > > > IMHO we should have a different approach for UI, different for Tasks,
> > > > different for Triggerer, and different for DagProcessor. (possibly
> the
> > > > Trigerer and DagProcessor could be the same because they share
> > > essentially
> > > > the same long-living token. Ash - I will add this to the discussion
> > > there.
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > > On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > Ah.. So if we are talking about a more complete approach - seeing
> > those
> > > > > comments from Ash - make me think if we should have another AIP.
> > > > > (connected) about splitting the distributions. We have never
> > finalized
> > > it
> > > > > (nor even discussed id) but Ash - you had some initial document for
> > > that.
> > > > > So maybe we should finalize it and rather than specify it in this
> > AIP -
> > > > > have a separate AIP about distribution split that AIP-92 could
> depend
> > > on.
> > > > > It seems much more reasonable to split "distribution and code
> split"
> > > from
> > > > > parsing isolation I think and implement them separately/in
> parallel.
> > > > >
> > > > > Reading Ash comments (and maybe I am going a bit further than Ash)
> it
> > > > > calls for something that I am a big proponent of - splitting
> > > > "airflow-core"
> > > > > and having a different "scheduler". "webserver", "dag processor"
> and
> > > > > "triggerer" distributions. Now - we have the capability of having
> > > > "shared"
> > > > > code - we do not need "common" code to make it happen - because we
> > can
> > > > > share code.
> > > > >
> > > > > What it could give us - on top of clean client/server split, we
> could
> > > > have
> > > > > different dependencies used by those distributions. Additionally,
> we
> > > > could
> > > > > also split-off the executors from providers and finally implement
> it
> > in
> > > > the
> > > > > way that scheduler does not use providers at all (not even
> > > > cncf.kubernetes
> > > > > nor celery providers installed in scheduler nor webserver but
> > > "executors"
> > > > > packages instead. The code sharing approach with symlinks we have
> now
> > > > will
> > > > > make it a .... breeze :) . That would also imply sharing
> "connection"
> > > > > definitions through DB, and likely implementing "test connection"
> > > > > feature properly finally (i.e executing test connection in worker /
> > > > > triggerer rather than in web server which is a reason why we
> disabled
> > > it
> > > > by
> > > > > default now). This way "api-server" would not need any of the
> > providers
> > > > to
> > > > > be installed either which IMHO is the biggest win from a security
> > point
> > > > of
> > > > > view.
> > > > >
> > > > > And the nice thing about it is that it would be rather transparent
> > when
> > > > > anyone uses "pip install apache-airflow" - it would behave exactly
> > the
> > > > > same, no more complexity involved, simply more distributions
> > installed
> > > > when
> > > > > 'apache-airflow" meta-distribution is used, but it would allow
> those
> > > who
> > > > > want to implement a more complex and secure setup to have different
> > > > > "environments" with modularized pieces of airflow installed - only
> > > > > "apache-airflow-dag-processor + task-sdk + providers" where
> > > dag-processor
> > > > > is run, only "apache-airflow-scheduler + executors" where scheduler
> > is
> > > > > installed only "apache-airflow-task-sdk + providers" where workers
> > are
> > > > > running, only "apache-airflow-api-server" where api-server is
> running
> > > and
> > > > > only "apache-airflow-trigger + task-sdk + providers" .
> > > > >
> > > > > I am happy (Ash If you are fine with that) to take that original
> > > document
> > > > > over and lead this part and new AIP to completion (including
> > > > > implementation), I am very much convinced that this will lead to
> much
> > > > > better dependency security and more modular code without impacting
> > the
> > > > > "apache-airflow" installation complexity.
> > > > >
> > > > > If we do it this way- the part of code/clean split would be
> > "delegated
> > > > > out" from AIP-92 to this new AIP and turned into dependency.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > > >
> > > > >> This AIP is definitely heading in the right direction and is a
> > feature
> > > > >> I’d like to see.
> > > > >>
> > > > >> For me the outstanding things that need more detail:
> > > > >>
> > > > >> 1. The Authentication token. How will this long lived token work
> > > without
> > > > >> being insecure. Who and what will generate it? How will we
> identify
> > > > >> top-level requests for Variables in order to be able to add
> Variable
> > > > >> RBAC/ACLs. This is an important enough thing that I think it needs
> > > > >> discussion before we vote on this AIP.
> > > > >> 2. Security generally — how will this work, especially with the
> > > > >> multi-team? I think this likely means making the APIs work on the
> > > bundle
> > > > >> level as you mention in the doc, but I haven’t thought deeply
> about
> > > this
> > > > >> yet.
> > > > >> 3. API Versioning? One of the the key driving goals with AIP-72
> and
> > > the
> > > > >> Task Execution SDK was the idea that “you can upgrade the API
> server
> > > as
> > > > you
> > > > >> like, and your clients/workers never need to work” — i.e. the API
> > > > server is
> > > > >> 100% working with all older versions of the TaskSDK. I don’t know
> if
> > > we
> > > > >> will achieve that goal in the long run but it is the desire, and
> > part
> > > of
> > > > >> why we are using CalVer and the Cadwyn library to provide API
> > > > versioning.
> > > > >> 4. As mentioned previously, not sure the existing serialised JSON
> > > format
> > > > >> for DAGs is correct, but since that now has version and we already
> > > have
> > > > the
> > > > >> ability to upgrade that somewhere in the Airflow Core that doesn’t
> > > > >> necessarily become a blocker/pre-requisite for this AIP.
> > > > >>
> > > > >> I think Dag parsing API client+submission+parsing process manager
> > > should
> > > > >> either live in the Task SDK dist, or in a new separate dist that
> > uses
> > > > >> TaskSDK, but crucially not in apache-airflow-core. My reason for
> > this
> > > is
> > > > >> that I want it to be possible for the server components
> (scheduler,
> > > API
> > > > >> server) to not need task-sdk installed (just for
> > cleanliness/avoiding
> > > > >> confusion about what versions it needs) and also vice-verse, to be
> > > able
> > > > to
> > > > >> run a “team worker bundle” (Dag parsing, workers, triggered/async
> > > > workers)
> > > > >> on whatever version of TaskSDK they choose, again without
> > > > >> apache-airflow-core installed for avoidance of doubt.
> > > > >>
> > > > >> Generally I would like this as it means we can have a nicer
> > separation
> > > > of
> > > > >> Core and Dag parsing code, as the dag parsing itself uses the SDK,
> > it
> > > > would
> > > > >> be nice to have a proper server/client split, both from a tighter
> > > > security
> > > > >> point-of-view, but also from a code layout point of view.
> > > > >>
> > > > >> -ash
> > > > >>
> > > > >>
> > > > >> > On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > > >> >
> > > > >> > Well, you started it - so it's up to you to decide if you think
> we
> > > > have
> > > > >> > consensus, or whether we need a vote.
> > > > >> >
> > > > >> > And It's not a question of "informal" vote but it's rather clear
> > > > >> following
> > > > >> > the https://www.apache.org/foundation/voting.html that we
> either
> > > > need a
> > > > >> > LAZY CONSENSUS or VOTE thread. Both are formal.
> > > > >> >
> > > > >> > This is the difficult part when you have a proposal, to assess
> (by
> > > > you)
> > > > >> > whether we are converging to consensus or whether vote is
> needed.
> > > > There
> > > > >> is
> > > > >> > no other body or "authority" to do it for you.
> > > > >> >
> > > > >> > J.
> > > > >> >
> > > > >> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <
> > > > sumeet.manit@gmail.com
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Sorry for nudging again, but can we get into some consensus on
> > > this?
> > > > I
> > > > >> mean
> > > > >> >> if this AIP isn't good enough, then we can drop it altogether
> and
> > > > >> someone
> > > > >> >> can rethink the whole thing. Should we do some kind of informal
> > > > voting
> > > > >> and
> > > > >> >> close this thread?
> > > > >> >>
> > > > >> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > > >> >>
> > > > >> >>>>> My main concern with this right now is the serialisation
> > format
> > > of
> > > > >> >> DAGs
> > > > >> >>> —
> > > > >> >>>> it wasn’t really designed with remote submission in mind, so
> it
> > > > need
> > > > >> >> some
> > > > >> >>>> careful examination to see if it is fit for this purpose or
> > not.
> > > > >> >>>
> > > > >> >>> I understand Ash's concerns - the format has not been designed
> > > with
> > > > >> >>> size/speed optimization in mind so **possibly** we could
> design
> > a
> > > > >> >> different
> > > > >> >>> format that would be better suited.
> > > > >> >>>
> > > > >> >>> BUT ... Done is better than perfect.
> > > > >> >>>
> > > > >> >>> I think there are a number of risks involved in changing the
> > > format
> > > > >> and
> > > > >> >> it
> > > > >> >>> could significantly increase time of development with
> uncertain
> > > > gains
> > > > >> at
> > > > >> >>> the end - also because of the progress in compression that
> > > happened
> > > > >> over
> > > > >> >>> the last few years.
> > > > >> >>>
> > > > >> >>> It might be a good idea to experiment a bit with different
> > > > compression
> > > > >> >>> algorithms for "our" dag representation and possibly we could
> > find
> > > > the
> > > > >> >> best
> > > > >> >>> algorithm for "airflow dag" type of json data. There are a lot
> > of
> > > > >> >>> repetitions in the JSON representation and I guess in "our"
> json
> > > > >> >>> representation there are some artifacts and repeated sections
> > that
> > > > >> simply
> > > > >> >>> might compress well with different algorithms. Also in this
> case
> > > > >> >>> speed matters (and CPU trade-off).
> > > > >> >>>
> > > > >> >>> Looking at compression "theory" - before we experiment with
> it -
> > > > >> there is
> > > > >> >>> the relatively new standard "zstandard"
> > > > >> https://github.com/facebook/zstd
> > > > >> >>> compression opensourced in 2016 which I've heard good things
> > > about -
> > > > >> >>> especially that it maintains a very good compression rate for
> > text
> > > > >> data,
> > > > >> >>> but also it is tremendously fast - especially for
> decompression
> > > > >> (which is
> > > > >> >>> super important factor for us - we compress new DAG
> > representation
> > > > far
> > > > >> >> less
> > > > >> >>> often than decompress it in general case). It is standardized
> in
> > > RFC
> > > > >> >>> https://datatracker.ietf.org/doc/html/rfc8878 and there are
> > > various
> > > > >> >>> implementations and it is even being added to Python standard
> > > > library
> > > > >> in
> > > > >> >>> Python 3.14
> > > > >> https://docs.python.org/3.14/library/compression.zstd.html
> > > > >> >> and
> > > > >> >>> there is a very well maintained python binding library
> > > > >> >>> https://pypi.org/project/zstd/ to Yann Collet (algorithm
> > author)
> > > > >> ZSTD C
> > > > >> >>> library. And libzstd is already part of our images - it is
> > needed
> > > by
> > > > >> >> other
> > > > >> >>> dependencies of ours. All with BSD licence, directly usable by
> > us.
> > > > >> >>>
> > > > >> >>> I think this one might be a good candidate for us to try, and
> > > > possibly
> > > > >> >> with
> > > > >> >>> zstd we could achieve both size and CPU overhead that would be
> > > > >> comparable
> > > > >> >>> with any "new" format we could come up with - especially that
> we
> > > are
> > > > >> >>> talking merely about processing a huge blob between "storable"
> > > > >> >> (compressed)
> > > > >> >>> and "locally usable" state (Python dict). We could likely use
> a
> > > > >> streaming
> > > > >> >>> JSON library (say the one that is used in Pydantic internally
> > > > >> >>> https://github.com/pydantic/jiter - we already have it as
> part
> > of
> > > > >> >>> Pydantic)
> > > > >> >>> to also save memory - we could stream decompressed stream into
> > > > jitter
> > > > >> so
> > > > >> >>> that both the json dict and string representation does not
> have
> > to
> > > > be
> > > > >> >>> loaded fully in memory at the same time. There are likely lots
> > of
> > > > >> >>> optimisations we could do - I mentioned possibly streaming the
> > > data
> > > > >> from
> > > > >> >>> API directly to DB (if this is possible - not sure)
> > > > >> >>>
> > > > >> >>> J.
> > > > >> >>>
> > > > >> >>>
> > > > >> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> > > > >> sumeet.manit@gmail.com>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >> >>>>>
> > > > >> >>>>> My main concern with this right now is the serialisation
> > format
> > > of
> > > > >> >>> DAGs —
> > > > >> >>>>> it wasn’t really designed with remote submission in mind, so
> > it
> > > > need
> > > > >> >>> some
> > > > >> >>>>> careful examination to see if it is fit for this purpose or
> > not.
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>> I'm not sure on this point, cause if we are able to convert a
> > DAG
> > > > >> into
> > > > >> >>>> JSON, then it has to be transferable over the internet.
> > > > >> >>>>
> > > > >> >>>> In particular One of the things I worry about is that the
> JSON
> > > can
> > > > >> get
> > > > >> >>> huge
> > > > >> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> Yeah, agree on this, thats why we can transfer compressed
> data
> > > > >> instead
> > > > >> >> of
> > > > >> >>>> real json. Of course, this won't guarantee that the payload
> > will
> > > > >> always
> > > > >> >>> be
> > > > >> >>>> small enough, but we can't say that it'll definitely happen
> > > either.
> > > > >> >>>>
> > > > >> >>>> I also wonder if as part of this proposal we should move the
> > > > Callback
> > > > >> >>>>> requests off the dag parsers and on to the workers instead
> > > > >> >>>>
> > > > >> >>>> let's make such a "workfload" implementation stream that
> could
> > > > >> support
> > > > >> >>> both
> > > > >> >>>>> - Deadlines and DAG parsing logic
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> I don't have any strong opinion here, but it feels like it's
> > > gonna
> > > > >> blow
> > > > >> >>> up
> > > > >> >>>> the scope of the AIP too much.
> > > > >> >>>>
> > > > >> >>>>
> > > > >> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <
> jarek@potiuk.com>
> > > > >> wrote:
> > > > >> >>>>
> > > > >> >>>>>> My main concern with this right now is the serialisation
> > format
> > > > of
> > > > >> >>>> DAGs —
> > > > >> >>>>> it wasn’t really designed with remote submission in mind, so
> > it
> > > > need
> > > > >> >>> some
> > > > >> >>>>> careful examination to see if it is fit for this purpose or
> > not.
> > > > >> >>>>>
> > > > >> >>>>> Yep. That might be potentially a problem (or at least "need
> > more
> > > > >> >>>> resources
> > > > >> >>>>> to run airflow") and that is where my "2x memory" came from
> if
> > > we
> > > > do
> > > > >> >> it
> > > > >> >>>> in
> > > > >> >>>>> a trivial way. Currently we a) keep the whole DAG in memory
> > when
> > > > >> >>>>> serializing it b) submit it to database (also using
> > essentially
> > > > some
> > > > >> >>> kind
> > > > >> >>>>> of API (implemented by the database client) - so we know the
> > > whole
> > > > >> >>> thing
> > > > >> >>>>> "might work" but indeed if you use a trivial implementation
> of
> > > > >> >>> submitting
> > > > >> >>>>> the whole json - it basically means that the whole json will
> > > have
> > > > to
> > > > >> >>> also
> > > > >> >>>>> be kept in the memory of API server. But we also compress it
> > > when
> > > > >> >>> needed
> > > > >> >>>> -
> > > > >> >>>>> I wonder what are the compression ratios we saw with those
> > > > 10-20MBs
> > > > >> >>> Dags
> > > > >> >>>> -
> > > > >> >>>>> if the problem is using strings where bool would suffice,
> > > > >> compression
> > > > >> >>>>> should generally help a lot. We could only ever send
> > compressed
> > > > data
> > > > >> >>> over
> > > > >> >>>>> the API - there seems to be no need to send "plain JSON"
> data
> > > over
> > > > >> >> the
> > > > >> >>>> API
> > > > >> >>>>> or storing the plain JSON in the DB (of course that trades
> > > memory
> > > > >> for
> > > > >> >>>> CPU).
> > > > >> >>>>>
> > > > >> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres)
> have
> > > > >> >> support
> > > > >> >>>> for
> > > > >> >>>>> any kind if binary data streaming - because that could help
> a
> > > lot
> > > > of
> > > > >> >> if
> > > > >> >>>> we
> > > > >> >>>>> could use streaming HTTP API and chunk and append the binary
> > > > chunks
> > > > >> >>> (when
> > > > >> >>>>> writing) - or read data in chunks ans stream them back via
> the
> > > > API.
> > > > >> >>> That
> > > > >> >>>>> could seriously decrease the amount of memory needed by the
> > API
> > > > >> >> server
> > > > >> >>> to
> > > > >> >>>>> process such huge serialized dags.
> > > > >> >>>>>
> > > > >> >>>>> And yeah - I would also love the "execute task" to be
> > > implemented
> > > > >> >> here
> > > > >> >>> -
> > > > >> >>>>> but I am not sure if this should be part of the same effort
> or
> > > > maybe
> > > > >> >> a
> > > > >> >>>>> separate implementation? That sounds very loosely coupled
> with
> > > DB
> > > > >> >>>>> isolation. And it seems a common theme - I think that would
> > also
> > > > >> make
> > > > >> >>> the
> > > > >> >>>>> sync Deadline alerts case that we discussed at the dev call
> > > > today. I
> > > > >> >>>> wonder
> > > > >> >>>>> if that should not be kind of parallel (let's make such a
> > > > >> "workfload"
> > > > >> >>>>> implementation stream that could support both - Deadlines
> and
> > > DAG
> > > > >> >>> parsing
> > > > >> >>>>> logic. We have already two "users" for it and I really love
> > the
> > > > >> >> saying
> > > > >> >>>> "if
> > > > >> >>>>> you want to make something reusable - make it usable
> first" -
> > > > seems
> > > > >> >>> like
> > > > >> >>>>> we might have good opportunity to make such workload
> > > > implementation
> > > > >> >>>> "doubly
> > > > >> >>>>> used" from the beginning which would increase chances it
> will
> > > be
> > > > >> >>>>> "reusable" for other things as well :).
> > > > >> >>>>>
> > > > >> >>>>> J.
> > > > >> >>>>>
> > > > >> >>>>>
> > > > >> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <
> > > > ash@apache.org>
> > > > >> >>>> wrote:
> > > > >> >>>>>
> > > > >> >>>>>> My main concern with this right now is the serialisation
> > format
> > > > of
> > > > >> >>>> DAGs —
> > > > >> >>>>>> it wasn’t really designed with remote submission in mind,
> so
> > it
> > > > >> >> need
> > > > >> >>>> some
> > > > >> >>>>>> careful examination to see if it is fit for this purpose or
> > > not.
> > > > >> >>>>>>
> > > > >> >>>>>> In particular One of the things I worry about is that the
> > JSON
> > > > can
> > > > >> >>> get
> > > > >> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!)
> > > > (which
> > > > >> >> is
> > > > >> >>>>>> likely due to things being included as text when a bool
> might
> > > > >> >>> suffice,
> > > > >> >>>>> for
> > > > >> >>>>>> example) But I don’t think “just submit the existing JSON
> > over
> > > an
> > > > >> >>> API”
> > > > >> >>>>> is a
> > > > >> >>>>>> good idea.
> > > > >> >>>>>>
> > > > >> >>>>>> I also wonder if as part of this proposal we should move
> the
> > > > >> >> Callback
> > > > >> >>>>>> requests off the dag parsers and on to the workers instead
> —
> > in
> > > > >> >>> AIP-72
> > > > >> >>>> we
> > > > >> >>>>>> introduced the concept of a Workload, with the only one
> > > existing
> > > > >> >>> right
> > > > >> >>>>> now
> > > > >> >>>>>> is “ExecuteTask”
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > > > >> >>>>>> — it might be time to finally move task and dag callbacks
> to
> > > the
> > > > >> >> same
> > > > >> >>>>> thing
> > > > >> >>>>>> and make dag parsers only responsible for, well, parsing.
> :)
> > > > >> >>>>>>
> > > > >> >>>>>> These are all solvable problems, and this will be a great
> > > feature
> > > > >> >> to
> > > > >> >>>>> have,
> > > > >> >>>>>> but we need to do some more thinking and planning first.
> > > > >> >>>>>>
> > > > >> >>>>>> -ash
> > > > >> >>>>>>
> > > > >> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> > > > >> >> sumeet.manit@gmail.com
> > > > >> >>>>
> > > > >> >>>>>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>> Gentle reminder for everyone to review the proposal.
> > > > >> >>>>>>>
> > > > >> >>>>>>> Updated link:
> > > > >> >>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > > > >> >>>>>>>
> > > > >> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > > > >> >>>>> sumeet.manit@gmail.com
> > > > >> >>>>>>>
> > > > >> >>>>>>> wrote:
> > > > >> >>>>>>>
> > > > >> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and
> others
> > > > >> >>>>> suggested, I
> > > > >> >>>>>>>> expanded the scope of this AIP and divided it into three
> > > > phases.
> > > > >> >>>> With
> > > > >> >>>>>> the
> > > > >> >>>>>>>> increased scope, the boundary line between this AIP and
> > > AIP-85
> > > > >> >>> got a
> > > > >> >>>>>> little
> > > > >> >>>>>>>> thinner, but I believe these are still two different
> > > > >> >> enhancements
> > > > >> >>> to
> > > > >> >>>>>> make.
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>
> > > > >> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > > >> >>>>>> sumeet.manit@gmail.com>
> > > > >> >>>>>>>> wrote:
> > > > >> >>>>>>>>
> > > > >> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well
> > to
> > > be
> > > > >> >>> part
> > > > >> >>>>> of
> > > > >> >>>>>>>>> this AIP and phase out the implementation. Though I
> didn't
> > > > >> >>> exclude
> > > > >> >>>>>> Triggers
> > > > >> >>>>>>>>> because "Uber" doesn't need that, I just thought of
> > keeping
> > > > the
> > > > >> >>>> scope
> > > > >> >>>>>> of
> > > > >> >>>>>>>>> development small and achieving them, just like it was
> > done
> > > in
> > > > >> >>>>> Airlfow
> > > > >> >>>>>> 3 by
> > > > >> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> But if you think Triggers should be part of this AIP
> > itself,
> > > > >> >>> then I
> > > > >> >>>>> can
> > > > >> >>>>>>>>> do that and include Triggers as well in it.
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <
> > > > jarek@potiuk.com
> > > > >> >>>
> > > > >> >>>>> wrote:
> > > > >> >>>>>>>>>
> > > > >> >>>>>>>>>> I would very much prefer the architectural choices of
> > this
> > > > AIP
> > > > >> >>> are
> > > > >> >>>>>> based
> > > > >> >>>>>>>>>> on
> > > > >> >>>>>>>>>> "general public" needs rather than "Uber needs" even if
> > > Uber
> > > > >> >>> will
> > > > >> >>>> be
> > > > >> >>>>>>>>>> implementing it - so from my point of view having
> Trigger
> > > > >> >>>> separation
> > > > >> >>>>>> as
> > > > >> >>>>>>>>>> part of it is quite important.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> But that's not even this.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> We've been discussing for example for Deadlines (being
> > > > >> >>> implemented
> > > > >> >>>>> by
> > > > >> >>>>>>>>>> Dennis and Ramit a possibility of short,
> > > notification-style
> > > > >> >>>>>> "deadlines"
> > > > >> >>>>>>>>>> to be send to triggerer for execution - this is well
> > > advanced
> > > > >> >>> now,
> > > > >> >>>>> and
> > > > >> >>>>>>>>>> whether you want it or not Dag-provided code might be
> > > > >> >> serialized
> > > > >> >>>> and
> > > > >> >>>>>> sent
> > > > >> >>>>>>>>>> to triggerer for execution. This is part of our
> "broader"
> > > > >> >>>>>> architectural
> > > > >> >>>>>>>>>> change where we treat "workers" and "triggerer"
> similarly
> > > as
> > > > a
> > > > >> >>>>> general
> > > > >> >>>>>>>>>> executors of "sync" and "async" tasks respectively.
> > That's
> > > > >> >> where
> > > > >> >>>>>> Airflow
> > > > >> >>>>>>>>>> is
> > > > >> >>>>>>>>>> evolving towards - inevitably.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> But we can of course phase things in out for
> > > implementation -
> > > > >> >>> even
> > > > >> >>>>> if
> > > > >> >>>>>> AIP
> > > > >> >>>>>>>>>> should cover both, I think if the goal of the AIP and
> > > > preamble
> > > > >> >>> is
> > > > >> >>>>>> about
> > > > >> >>>>>>>>>> separating "user code" from "database" as the main
> > reason,
> > > it
> > > > >> >>> also
> > > > >> >>>>>> means
> > > > >> >>>>>>>>>> Triggerer if you ask me (from design point of view at
> > > least).
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> Again implementation can be phased and even different
> > > people
> > > > >> >> and
> > > > >> >>>>> teams
> > > > >> >>>>>>>>>> might work on those phases/pieces.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> J.
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > > >> >>>>>> sumeet.manit@gmail.com
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> > triggerers
> > > > as
> > > > >> >>>> well,
> > > > >> >>>>>>>>>> but
> > > > >> >>>>>>>>>>>> that
> > > > >> >>>>>>>>>>>> can be done as part of a different AIP
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> You won't achieve your goal of "true" isolation of
> user
> > > code
> > > > >> >> if
> > > > >> >>>> you
> > > > >> >>>>>>>>>> don't
> > > > >> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it -
> it
> > > > >> >> should
> > > > >> >>>>> cover
> > > > >> >>>>>>>>>>> both.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> My bad, should've explained our architecture for
> > triggers
> > > as
> > > > >> >>>> well,
> > > > >> >>>>>>>>>>> apologies. So here it is:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> - Triggers would be running on a centralized service,
> > so
> > > > >> >> all
> > > > >> >>>> the
> > > > >> >>>>>>>>>> Trigger
> > > > >> >>>>>>>>>>> classes will be part of the platform team's repo and
> > not
> > > > >> >> the
> > > > >> >>>>>>>>>> customer's
> > > > >> >>>>>>>>>>> repo
> > > > >> >>>>>>>>>>> - The triggers won't be able to use any libs other
> than
> > > std
> > > > >> >>>> ones,
> > > > >> >>>>>>>>>> which
> > > > >> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> > > > >> >>>>>>>>>>> - As we are the owners of the core Airflow repo,
> > > customers
> > > > >> >>> have
> > > > >> >>>>> to
> > > > >> >>>>>>>>>> get
> > > > >> >>>>>>>>>>> our approval to land any class in this path (unlike
> the
> > > > >> >> dags
> > > > >> >>>> repo
> > > > >> >>>>>>>>>> which
> > > > >> >>>>>>>>>>> they own)
> > > > >> >>>>>>>>>>> - When a customer's task defer, we would have an
> > > allowlist
> > > > >> >> on
> > > > >> >>>> our
> > > > >> >>>>>>>>>> side
> > > > >> >>>>>>>>>>> to check if we should do the async polling or not
> > > > >> >>>>>>>>>>> - If the Trigger class isn't part of our repo
> > > (allowlist),
> > > > >> >>> just
> > > > >> >>>>>>>>>> fail the
> > > > >> >>>>>>>>>>> task, as anyway we won't be having the code that they
> > > used
> > > > >> >> in
> > > > >> >>>> the
> > > > >> >>>>>>>>>>> trigger
> > > > >> >>>>>>>>>>> class
> > > > >> >>>>>>>>>>> - If any of these conditions aren't suitable for you
> > (as
> > > a
> > > > >> >>>>>>>>>> customer),
> > > > >> >>>>>>>>>>> feel free to use sync tasks only
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> > > > >> >> communicate
> > > > >> >>>> over
> > > > >> >>>>>>>>>> apis
> > > > >> >>>>>>>>>>> only. If that is done, then we can have instances of
> > > > >> >> triggerer
> > > > >> >>>> svc
> > > > >> >>>>>>>>>> running
> > > > >> >>>>>>>>>>> at customer's side as well, which can process any type
> > of
> > > > >> >>> trigger
> > > > >> >>>>>>>>>> class.
> > > > >> >>>>>>>>>>> Though that's not a blocker for us at the moment,
> cause
> > > > >> >>> triggerer
> > > > >> >>>>> are
> > > > >> >>>>>>>>>>> mostly doing just polling using simple libs like
> > requests.
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > > >> >>>>>>>>>> <ikholopov@google.com.invalid
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I
> > believe
> > > > it
> > > > >> >>>>> aligns
> > > > >> >>>>>>>>>> well
> > > > >> >>>>>>>>>>>> with the goals of making Airflow well-scalable
> beyond a
> > > > >> >>>>> single-team
> > > > >> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full
> > > support
> > > > >> >>> with
> > > > >> >>>>> this
> > > > >> >>>>>>>>>> one.
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> I've left a couple of clarification requests on the
> AIP
> > > > >> >> page.
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> Thanks,
> > > > >> >>>>>>>>>>>> Igor
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > > >> >>>>>>>>>>> sumeet.manit@gmail.com>
> > > > >> >>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's
> > good
> > > to
> > > > >> >>> know
> > > > >> >>>>>>>>>> that
> > > > >> >>>>>>>>>>> the
> > > > >> >>>>>>>>>>>>> DAG processor has some preemptive measures in place
> to
> > > > >> >>> prevent
> > > > >> >>>>>>>>>> access
> > > > >> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to
> > > solve
> > > > >> >> is
> > > > >> >>>> not
> > > > >> >>>>> to
> > > > >> >>>>>>>>>>>> provide
> > > > >> >>>>>>>>>>>>> DB creds to the customer teams, who are using
> Airflow
> > > as a
> > > > >> >>>>>>>>>> multi-tenant
> > > > >> >>>>>>>>>>>>> orchestration platform. I've updated the doc to
> > reflect
> > > > >> >> this
> > > > >> >>>>> point
> > > > >> >>>>>>>>>> as
> > > > >> >>>>>>>>>>>> well.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Answering Jarek's points,
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism,
> > > added
> > > > >> >>> that
> > > > >> >>>>> in
> > > > >> >>>>>>>>>> doc,
> > > > >> >>>>>>>>>>>> but
> > > > >> >>>>>>>>>>>>> still how the token can be obtained (safely) is
> still
> > > open
> > > > >> >> in
> > > > >> >>>> my
> > > > >> >>>>>>>>>> mind.
> > > > >> >>>>>>>>>>> I
> > > > >> >>>>>>>>>>>>> believe the token used by task executors can be
> > created
> > > > >> >>> outside
> > > > >> >>>>> of
> > > > >> >>>>>>>>>> it
> > > > >> >>>>>>>>>>> as
> > > > >> >>>>>>>>>>>>> well (I may be wrong here).
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> > triggerers
> > > > as
> > > > >> >>>> well,
> > > > >> >>>>>>>>>> but
> > > > >> >>>>>>>>>>>> that
> > > > >> >>>>>>>>>>>>> can be done as part of a different AIP
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work
> largely.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs
> we
> > > can
> > > > >> >>> work
> > > > >> >>>>>>>>>> with
> > > > >> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would
> be
> > > > >> >> treated
> > > > >> >>>> as
> > > > >> >>>>> a
> > > > >> >>>>>>>>>> diff
> > > > >> >>>>>>>>>>>>> bundle.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> Also, added points around callbacks, as these are
> also
> > > > >> >>> fetched
> > > > >> >>>>>>>>>> directly
> > > > >> >>>>>>>>>>>>> from the DB.
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > > > >> >>>> jarek@potiuk.com>
> > > > >> >>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> > > likely
> > > > >> >>> not
> > > > >> >>>>>>>>>>>> protection
> > > > >> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> does
> > > > >> >>> protect
> > > > >> >>>>>>>>>> against
> > > > >> >>>>>>>>>>>>>> casual DB access attempts - the db session is
> blanked
> > > out
> > > > >> >> in
> > > > >> >>>> the
> > > > >> >>>>>>>>>>>> parsing
> > > > >> >>>>>>>>>>>>>> process , as are the env var configs
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > >> >>>>>>>>>>>>>> -
> > > > >> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no
> > > protection
> > > > >> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed
> > back
> > > > >> >> then
> > > > >> >>> in
> > > > >> >>>>>>>>>> March
> > > > >> >>>>>>>>>>> I
> > > > >> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with
> > full
> > > > >> >>>> knowledge
> > > > >> >>>>>>>>>> it's
> > > > >> >>>>>>>>>>>> not
> > > > >> >>>>>>>>>>>>>> protecting against all threats.
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > > > >> >>>>>>>>>> ash@apache.org>
> > > > >> >>>>>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> > > likely
> > > > >> >>> not
> > > > >> >>>>>>>>>>>> protection
> > > > >> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> > does
> > > > >> >>> protect
> > > > >> >>>>>>>>>>> against
> > > > >> >>>>>>>>>>>>>>> casual DB access attempts - the db session is
> > blanked
> > > > out
> > > > >> >>> in
> > > > >> >>>>>>>>>> the
> > > > >> >>>>>>>>>>>>> parsing
> > > > >> >>>>>>>>>>>>>>> process , as are the env var configs
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > >> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> > > > >> >> protection
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> > > > >> >> jarek@potiuk.com>
> > > > >> >>>>>>>>>> wrote:
> > > > >> >>>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no
> > > > built-in
> > > > >> >>>>>>>>>>> protection
> > > > >> >>>>>>>>>>>>>>> against
> > > > >> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example -
> read
> > > > >> >>> database
> > > > >> >>>>>>>>>>>>>>>> credentials from airflow configuration and use
> them
> > > to
> > > > >> >>> talk
> > > > >> >>>>>>>>>> to DB
> > > > >> >>>>>>>>>>>>>>> directly.
> > > > >> >>>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>>
> > > > >> >>>>>>>>>>>>
> > > > >> >>>>>>>>>>>
> > > > >> >>>>>>>>>>
> > > > >> >>>>>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>>
> > > > >> >>>>>
> > > > >> >>>>
> > > > >> >>>
> > > > >> >>
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Awesome, having a standard approach for all kinds of authentication would
be great, looking forward to it.
BTW, on a side note, I see that as of now, things like Connections,
Variables, and XComs which are present under Execution API namespace, don't
have any authorization model (left with TODOs), so is there any plan how
they will work, cause we might need something similar for
Dag-processor/Triggerer as well.
Also, as we've decided to create a diff API namespace and not use execution
namespace for hosting new APIs required by dag-processor/triggerer, do
we've to copy these classes & routes OR atleast refactor them, so they can
serve all API namespaces.
On Wed, Aug 13, 2025 at 1:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Will do. We are also discussing - for now - within the security team - the
> various aspects of authentication approach we want to have - for both
> "UI/User authentication" as well as "Long running services" and how they
> relate to token exchange, invalidation and other scenarios. What we will
> come up - I hope shortly - is a proposal of general "model" of all kinds of
> security and authentication scenarios, so that we do not have to reinvent
> the wheel and try to figure out all the aspects with individual AIPs. We
> are not far from bringing it to the open discussion at devlist, but we have
> to be careful about some of the aspects that we might need to improve in
> the current setup to close some - small - loopholes so bear with us :)
>
> On Wed, Aug 13, 2025 at 9:33 AM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
> > Thanks Ash and Jarek, for the detailed comments. I largely agree with the
> > points mentioned by you guys, hence I updated the AIP and added a section
> > on Authentication, API Versioning, and Packaging as well. Please go
> through
> > it once more and let me know if there are more things to consider before
> I
> > open it for voting.
> >
> >
> > On Thu, Aug 7, 2025 at 6:03 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Also
> > >
> > > > 1. The Authentication token. How will this long lived token work
> > without
> > > being insecure. Who and what will generate it? How will we identify
> > > top-level requests for Variables in order to be able to add Variable
> > > RBAC/ACLs. This is an important enough thing that I think it needs
> > > discussion before we vote on this AIP.
> > >
> > > We are currently discussing - in the security team - approach for JWT
> > token
> > > handling, so likely we could move the discussion there, it does have
> some
> > > security implications and I think we should bring our finding to the
> > > devlist when we complete it, but I think we should add this case there.
> > > IMHO we should have a different approach for UI, different for Tasks,
> > > different for Triggerer, and different for DagProcessor. (possibly the
> > > Trigerer and DagProcessor could be the same because they share
> > essentially
> > > the same long-living token. Ash - I will add this to the discussion
> > there.
> > >
> > > J.
> > >
> > >
> > >
> > > On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Ah.. So if we are talking about a more complete approach - seeing
> those
> > > > comments from Ash - make me think if we should have another AIP.
> > > > (connected) about splitting the distributions. We have never
> finalized
> > it
> > > > (nor even discussed id) but Ash - you had some initial document for
> > that.
> > > > So maybe we should finalize it and rather than specify it in this
> AIP -
> > > > have a separate AIP about distribution split that AIP-92 could depend
> > on.
> > > > It seems much more reasonable to split "distribution and code split"
> > from
> > > > parsing isolation I think and implement them separately/in parallel.
> > > >
> > > > Reading Ash comments (and maybe I am going a bit further than Ash) it
> > > > calls for something that I am a big proponent of - splitting
> > > "airflow-core"
> > > > and having a different "scheduler". "webserver", "dag processor" and
> > > > "triggerer" distributions. Now - we have the capability of having
> > > "shared"
> > > > code - we do not need "common" code to make it happen - because we
> can
> > > > share code.
> > > >
> > > > What it could give us - on top of clean client/server split, we could
> > > have
> > > > different dependencies used by those distributions. Additionally, we
> > > could
> > > > also split-off the executors from providers and finally implement it
> in
> > > the
> > > > way that scheduler does not use providers at all (not even
> > > cncf.kubernetes
> > > > nor celery providers installed in scheduler nor webserver but
> > "executors"
> > > > packages instead. The code sharing approach with symlinks we have now
> > > will
> > > > make it a .... breeze :) . That would also imply sharing "connection"
> > > > definitions through DB, and likely implementing "test connection"
> > > > feature properly finally (i.e executing test connection in worker /
> > > > triggerer rather than in web server which is a reason why we disabled
> > it
> > > by
> > > > default now). This way "api-server" would not need any of the
> providers
> > > to
> > > > be installed either which IMHO is the biggest win from a security
> point
> > > of
> > > > view.
> > > >
> > > > And the nice thing about it is that it would be rather transparent
> when
> > > > anyone uses "pip install apache-airflow" - it would behave exactly
> the
> > > > same, no more complexity involved, simply more distributions
> installed
> > > when
> > > > 'apache-airflow" meta-distribution is used, but it would allow those
> > who
> > > > want to implement a more complex and secure setup to have different
> > > > "environments" with modularized pieces of airflow installed - only
> > > > "apache-airflow-dag-processor + task-sdk + providers" where
> > dag-processor
> > > > is run, only "apache-airflow-scheduler + executors" where scheduler
> is
> > > > installed only "apache-airflow-task-sdk + providers" where workers
> are
> > > > running, only "apache-airflow-api-server" where api-server is running
> > and
> > > > only "apache-airflow-trigger + task-sdk + providers" .
> > > >
> > > > I am happy (Ash If you are fine with that) to take that original
> > document
> > > > over and lead this part and new AIP to completion (including
> > > > implementation), I am very much convinced that this will lead to much
> > > > better dependency security and more modular code without impacting
> the
> > > > "apache-airflow" installation complexity.
> > > >
> > > > If we do it this way- the part of code/clean split would be
> "delegated
> > > > out" from AIP-92 to this new AIP and turned into dependency.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > > >
> > > >> This AIP is definitely heading in the right direction and is a
> feature
> > > >> I’d like to see.
> > > >>
> > > >> For me the outstanding things that need more detail:
> > > >>
> > > >> 1. The Authentication token. How will this long lived token work
> > without
> > > >> being insecure. Who and what will generate it? How will we identify
> > > >> top-level requests for Variables in order to be able to add Variable
> > > >> RBAC/ACLs. This is an important enough thing that I think it needs
> > > >> discussion before we vote on this AIP.
> > > >> 2. Security generally — how will this work, especially with the
> > > >> multi-team? I think this likely means making the APIs work on the
> > bundle
> > > >> level as you mention in the doc, but I haven’t thought deeply about
> > this
> > > >> yet.
> > > >> 3. API Versioning? One of the the key driving goals with AIP-72 and
> > the
> > > >> Task Execution SDK was the idea that “you can upgrade the API server
> > as
> > > you
> > > >> like, and your clients/workers never need to work” — i.e. the API
> > > server is
> > > >> 100% working with all older versions of the TaskSDK. I don’t know if
> > we
> > > >> will achieve that goal in the long run but it is the desire, and
> part
> > of
> > > >> why we are using CalVer and the Cadwyn library to provide API
> > > versioning.
> > > >> 4. As mentioned previously, not sure the existing serialised JSON
> > format
> > > >> for DAGs is correct, but since that now has version and we already
> > have
> > > the
> > > >> ability to upgrade that somewhere in the Airflow Core that doesn’t
> > > >> necessarily become a blocker/pre-requisite for this AIP.
> > > >>
> > > >> I think Dag parsing API client+submission+parsing process manager
> > should
> > > >> either live in the Task SDK dist, or in a new separate dist that
> uses
> > > >> TaskSDK, but crucially not in apache-airflow-core. My reason for
> this
> > is
> > > >> that I want it to be possible for the server components (scheduler,
> > API
> > > >> server) to not need task-sdk installed (just for
> cleanliness/avoiding
> > > >> confusion about what versions it needs) and also vice-verse, to be
> > able
> > > to
> > > >> run a “team worker bundle” (Dag parsing, workers, triggered/async
> > > workers)
> > > >> on whatever version of TaskSDK they choose, again without
> > > >> apache-airflow-core installed for avoidance of doubt.
> > > >>
> > > >> Generally I would like this as it means we can have a nicer
> separation
> > > of
> > > >> Core and Dag parsing code, as the dag parsing itself uses the SDK,
> it
> > > would
> > > >> be nice to have a proper server/client split, both from a tighter
> > > security
> > > >> point-of-view, but also from a code layout point of view.
> > > >>
> > > >> -ash
> > > >>
> > > >>
> > > >> > On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >> >
> > > >> > Well, you started it - so it's up to you to decide if you think we
> > > have
> > > >> > consensus, or whether we need a vote.
> > > >> >
> > > >> > And It's not a question of "informal" vote but it's rather clear
> > > >> following
> > > >> > the https://www.apache.org/foundation/voting.html that we either
> > > need a
> > > >> > LAZY CONSENSUS or VOTE thread. Both are formal.
> > > >> >
> > > >> > This is the difficult part when you have a proposal, to assess (by
> > > you)
> > > >> > whether we are converging to consensus or whether vote is needed.
> > > There
> > > >> is
> > > >> > no other body or "authority" to do it for you.
> > > >> >
> > > >> > J.
> > > >> >
> > > >> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <
> > > sumeet.manit@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> >> Sorry for nudging again, but can we get into some consensus on
> > this?
> > > I
> > > >> mean
> > > >> >> if this AIP isn't good enough, then we can drop it altogether and
> > > >> someone
> > > >> >> can rethink the whole thing. Should we do some kind of informal
> > > voting
> > > >> and
> > > >> >> close this thread?
> > > >> >>
> > > >> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > >> >>
> > > >> >>>>> My main concern with this right now is the serialisation
> format
> > of
> > > >> >> DAGs
> > > >> >>> —
> > > >> >>>> it wasn’t really designed with remote submission in mind, so it
> > > need
> > > >> >> some
> > > >> >>>> careful examination to see if it is fit for this purpose or
> not.
> > > >> >>>
> > > >> >>> I understand Ash's concerns - the format has not been designed
> > with
> > > >> >>> size/speed optimization in mind so **possibly** we could design
> a
> > > >> >> different
> > > >> >>> format that would be better suited.
> > > >> >>>
> > > >> >>> BUT ... Done is better than perfect.
> > > >> >>>
> > > >> >>> I think there are a number of risks involved in changing the
> > format
> > > >> and
> > > >> >> it
> > > >> >>> could significantly increase time of development with uncertain
> > > gains
> > > >> at
> > > >> >>> the end - also because of the progress in compression that
> > happened
> > > >> over
> > > >> >>> the last few years.
> > > >> >>>
> > > >> >>> It might be a good idea to experiment a bit with different
> > > compression
> > > >> >>> algorithms for "our" dag representation and possibly we could
> find
> > > the
> > > >> >> best
> > > >> >>> algorithm for "airflow dag" type of json data. There are a lot
> of
> > > >> >>> repetitions in the JSON representation and I guess in "our" json
> > > >> >>> representation there are some artifacts and repeated sections
> that
> > > >> simply
> > > >> >>> might compress well with different algorithms. Also in this case
> > > >> >>> speed matters (and CPU trade-off).
> > > >> >>>
> > > >> >>> Looking at compression "theory" - before we experiment with it -
> > > >> there is
> > > >> >>> the relatively new standard "zstandard"
> > > >> https://github.com/facebook/zstd
> > > >> >>> compression opensourced in 2016 which I've heard good things
> > about -
> > > >> >>> especially that it maintains a very good compression rate for
> text
> > > >> data,
> > > >> >>> but also it is tremendously fast - especially for decompression
> > > >> (which is
> > > >> >>> super important factor for us - we compress new DAG
> representation
> > > far
> > > >> >> less
> > > >> >>> often than decompress it in general case). It is standardized in
> > RFC
> > > >> >>> https://datatracker.ietf.org/doc/html/rfc8878 and there are
> > various
> > > >> >>> implementations and it is even being added to Python standard
> > > library
> > > >> in
> > > >> >>> Python 3.14
> > > >> https://docs.python.org/3.14/library/compression.zstd.html
> > > >> >> and
> > > >> >>> there is a very well maintained python binding library
> > > >> >>> https://pypi.org/project/zstd/ to Yann Collet (algorithm
> author)
> > > >> ZSTD C
> > > >> >>> library. And libzstd is already part of our images - it is
> needed
> > by
> > > >> >> other
> > > >> >>> dependencies of ours. All with BSD licence, directly usable by
> us.
> > > >> >>>
> > > >> >>> I think this one might be a good candidate for us to try, and
> > > possibly
> > > >> >> with
> > > >> >>> zstd we could achieve both size and CPU overhead that would be
> > > >> comparable
> > > >> >>> with any "new" format we could come up with - especially that we
> > are
> > > >> >>> talking merely about processing a huge blob between "storable"
> > > >> >> (compressed)
> > > >> >>> and "locally usable" state (Python dict). We could likely use a
> > > >> streaming
> > > >> >>> JSON library (say the one that is used in Pydantic internally
> > > >> >>> https://github.com/pydantic/jiter - we already have it as part
> of
> > > >> >>> Pydantic)
> > > >> >>> to also save memory - we could stream decompressed stream into
> > > jitter
> > > >> so
> > > >> >>> that both the json dict and string representation does not have
> to
> > > be
> > > >> >>> loaded fully in memory at the same time. There are likely lots
> of
> > > >> >>> optimisations we could do - I mentioned possibly streaming the
> > data
> > > >> from
> > > >> >>> API directly to DB (if this is possible - not sure)
> > > >> >>>
> > > >> >>> J.
> > > >> >>>
> > > >> >>>
> > > >> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> > > >> sumeet.manit@gmail.com>
> > > >> >>> wrote:
> > > >> >>>
> > > >> >>>>>
> > > >> >>>>> My main concern with this right now is the serialisation
> format
> > of
> > > >> >>> DAGs —
> > > >> >>>>> it wasn’t really designed with remote submission in mind, so
> it
> > > need
> > > >> >>> some
> > > >> >>>>> careful examination to see if it is fit for this purpose or
> not.
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>> I'm not sure on this point, cause if we are able to convert a
> DAG
> > > >> into
> > > >> >>>> JSON, then it has to be transferable over the internet.
> > > >> >>>>
> > > >> >>>> In particular One of the things I worry about is that the JSON
> > can
> > > >> get
> > > >> >>> huge
> > > >> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> Yeah, agree on this, thats why we can transfer compressed data
> > > >> instead
> > > >> >> of
> > > >> >>>> real json. Of course, this won't guarantee that the payload
> will
> > > >> always
> > > >> >>> be
> > > >> >>>> small enough, but we can't say that it'll definitely happen
> > either.
> > > >> >>>>
> > > >> >>>> I also wonder if as part of this proposal we should move the
> > > Callback
> > > >> >>>>> requests off the dag parsers and on to the workers instead
> > > >> >>>>
> > > >> >>>> let's make such a "workfload" implementation stream that could
> > > >> support
> > > >> >>> both
> > > >> >>>>> - Deadlines and DAG parsing logic
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> I don't have any strong opinion here, but it feels like it's
> > gonna
> > > >> blow
> > > >> >>> up
> > > >> >>>> the scope of the AIP too much.
> > > >> >>>>
> > > >> >>>>
> > > >> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com>
> > > >> wrote:
> > > >> >>>>
> > > >> >>>>>> My main concern with this right now is the serialisation
> format
> > > of
> > > >> >>>> DAGs —
> > > >> >>>>> it wasn’t really designed with remote submission in mind, so
> it
> > > need
> > > >> >>> some
> > > >> >>>>> careful examination to see if it is fit for this purpose or
> not.
> > > >> >>>>>
> > > >> >>>>> Yep. That might be potentially a problem (or at least "need
> more
> > > >> >>>> resources
> > > >> >>>>> to run airflow") and that is where my "2x memory" came from if
> > we
> > > do
> > > >> >> it
> > > >> >>>> in
> > > >> >>>>> a trivial way. Currently we a) keep the whole DAG in memory
> when
> > > >> >>>>> serializing it b) submit it to database (also using
> essentially
> > > some
> > > >> >>> kind
> > > >> >>>>> of API (implemented by the database client) - so we know the
> > whole
> > > >> >>> thing
> > > >> >>>>> "might work" but indeed if you use a trivial implementation of
> > > >> >>> submitting
> > > >> >>>>> the whole json - it basically means that the whole json will
> > have
> > > to
> > > >> >>> also
> > > >> >>>>> be kept in the memory of API server. But we also compress it
> > when
> > > >> >>> needed
> > > >> >>>> -
> > > >> >>>>> I wonder what are the compression ratios we saw with those
> > > 10-20MBs
> > > >> >>> Dags
> > > >> >>>> -
> > > >> >>>>> if the problem is using strings where bool would suffice,
> > > >> compression
> > > >> >>>>> should generally help a lot. We could only ever send
> compressed
> > > data
> > > >> >>> over
> > > >> >>>>> the API - there seems to be no need to send "plain JSON" data
> > over
> > > >> >> the
> > > >> >>>> API
> > > >> >>>>> or storing the plain JSON in the DB (of course that trades
> > memory
> > > >> for
> > > >> >>>> CPU).
> > > >> >>>>>
> > > >> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
> > > >> >> support
> > > >> >>>> for
> > > >> >>>>> any kind if binary data streaming - because that could help a
> > lot
> > > of
> > > >> >> if
> > > >> >>>> we
> > > >> >>>>> could use streaming HTTP API and chunk and append the binary
> > > chunks
> > > >> >>> (when
> > > >> >>>>> writing) - or read data in chunks ans stream them back via the
> > > API.
> > > >> >>> That
> > > >> >>>>> could seriously decrease the amount of memory needed by the
> API
> > > >> >> server
> > > >> >>> to
> > > >> >>>>> process such huge serialized dags.
> > > >> >>>>>
> > > >> >>>>> And yeah - I would also love the "execute task" to be
> > implemented
> > > >> >> here
> > > >> >>> -
> > > >> >>>>> but I am not sure if this should be part of the same effort or
> > > maybe
> > > >> >> a
> > > >> >>>>> separate implementation? That sounds very loosely coupled with
> > DB
> > > >> >>>>> isolation. And it seems a common theme - I think that would
> also
> > > >> make
> > > >> >>> the
> > > >> >>>>> sync Deadline alerts case that we discussed at the dev call
> > > today. I
> > > >> >>>> wonder
> > > >> >>>>> if that should not be kind of parallel (let's make such a
> > > >> "workfload"
> > > >> >>>>> implementation stream that could support both - Deadlines and
> > DAG
> > > >> >>> parsing
> > > >> >>>>> logic. We have already two "users" for it and I really love
> the
> > > >> >> saying
> > > >> >>>> "if
> > > >> >>>>> you want to make something reusable - make it usable first" -
> > > seems
> > > >> >>> like
> > > >> >>>>> we might have good opportunity to make such workload
> > > implementation
> > > >> >>>> "doubly
> > > >> >>>>> used" from the beginning which would increase chances it will
> > be
> > > >> >>>>> "reusable" for other things as well :).
> > > >> >>>>>
> > > >> >>>>> J.
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <
> > > ash@apache.org>
> > > >> >>>> wrote:
> > > >> >>>>>
> > > >> >>>>>> My main concern with this right now is the serialisation
> format
> > > of
> > > >> >>>> DAGs —
> > > >> >>>>>> it wasn’t really designed with remote submission in mind, so
> it
> > > >> >> need
> > > >> >>>> some
> > > >> >>>>>> careful examination to see if it is fit for this purpose or
> > not.
> > > >> >>>>>>
> > > >> >>>>>> In particular One of the things I worry about is that the
> JSON
> > > can
> > > >> >>> get
> > > >> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!)
> > > (which
> > > >> >> is
> > > >> >>>>>> likely due to things being included as text when a bool might
> > > >> >>> suffice,
> > > >> >>>>> for
> > > >> >>>>>> example) But I don’t think “just submit the existing JSON
> over
> > an
> > > >> >>> API”
> > > >> >>>>> is a
> > > >> >>>>>> good idea.
> > > >> >>>>>>
> > > >> >>>>>> I also wonder if as part of this proposal we should move the
> > > >> >> Callback
> > > >> >>>>>> requests off the dag parsers and on to the workers instead —
> in
> > > >> >>> AIP-72
> > > >> >>>> we
> > > >> >>>>>> introduced the concept of a Workload, with the only one
> > existing
> > > >> >>> right
> > > >> >>>>> now
> > > >> >>>>>> is “ExecuteTask”
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > > >> >>>>>> — it might be time to finally move task and dag callbacks to
> > the
> > > >> >> same
> > > >> >>>>> thing
> > > >> >>>>>> and make dag parsers only responsible for, well, parsing. :)
> > > >> >>>>>>
> > > >> >>>>>> These are all solvable problems, and this will be a great
> > feature
> > > >> >> to
> > > >> >>>>> have,
> > > >> >>>>>> but we need to do some more thinking and planning first.
> > > >> >>>>>>
> > > >> >>>>>> -ash
> > > >> >>>>>>
> > > >> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> > > >> >> sumeet.manit@gmail.com
> > > >> >>>>
> > > >> >>>>>> wrote:
> > > >> >>>>>>>
> > > >> >>>>>>> Gentle reminder for everyone to review the proposal.
> > > >> >>>>>>>
> > > >> >>>>>>> Updated link:
> > > >> >>>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > > >> >>>>>>>
> > > >> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > > >> >>>>> sumeet.manit@gmail.com
> > > >> >>>>>>>
> > > >> >>>>>>> wrote:
> > > >> >>>>>>>
> > > >> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and others
> > > >> >>>>> suggested, I
> > > >> >>>>>>>> expanded the scope of this AIP and divided it into three
> > > phases.
> > > >> >>>> With
> > > >> >>>>>> the
> > > >> >>>>>>>> increased scope, the boundary line between this AIP and
> > AIP-85
> > > >> >>> got a
> > > >> >>>>>> little
> > > >> >>>>>>>> thinner, but I believe these are still two different
> > > >> >> enhancements
> > > >> >>> to
> > > >> >>>>>> make.
> > > >> >>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>>
> > > >> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > >> >>>>>> sumeet.manit@gmail.com>
> > > >> >>>>>>>> wrote:
> > > >> >>>>>>>>
> > > >> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well
> to
> > be
> > > >> >>> part
> > > >> >>>>> of
> > > >> >>>>>>>>> this AIP and phase out the implementation. Though I didn't
> > > >> >>> exclude
> > > >> >>>>>> Triggers
> > > >> >>>>>>>>> because "Uber" doesn't need that, I just thought of
> keeping
> > > the
> > > >> >>>> scope
> > > >> >>>>>> of
> > > >> >>>>>>>>> development small and achieving them, just like it was
> done
> > in
> > > >> >>>>> Airlfow
> > > >> >>>>>> 3 by
> > > >> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> But if you think Triggers should be part of this AIP
> itself,
> > > >> >>> then I
> > > >> >>>>> can
> > > >> >>>>>>>>> do that and include Triggers as well in it.
> > > >> >>>>>>>>>
> > > >> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <
> > > jarek@potiuk.com
> > > >> >>>
> > > >> >>>>> wrote:
> > > >> >>>>>>>>>
> > > >> >>>>>>>>>> I would very much prefer the architectural choices of
> this
> > > AIP
> > > >> >>> are
> > > >> >>>>>> based
> > > >> >>>>>>>>>> on
> > > >> >>>>>>>>>> "general public" needs rather than "Uber needs" even if
> > Uber
> > > >> >>> will
> > > >> >>>> be
> > > >> >>>>>>>>>> implementing it - so from my point of view having Trigger
> > > >> >>>> separation
> > > >> >>>>>> as
> > > >> >>>>>>>>>> part of it is quite important.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> But that's not even this.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> We've been discussing for example for Deadlines (being
> > > >> >>> implemented
> > > >> >>>>> by
> > > >> >>>>>>>>>> Dennis and Ramit a possibility of short,
> > notification-style
> > > >> >>>>>> "deadlines"
> > > >> >>>>>>>>>> to be send to triggerer for execution - this is well
> > advanced
> > > >> >>> now,
> > > >> >>>>> and
> > > >> >>>>>>>>>> whether you want it or not Dag-provided code might be
> > > >> >> serialized
> > > >> >>>> and
> > > >> >>>>>> sent
> > > >> >>>>>>>>>> to triggerer for execution. This is part of our "broader"
> > > >> >>>>>> architectural
> > > >> >>>>>>>>>> change where we treat "workers" and "triggerer" similarly
> > as
> > > a
> > > >> >>>>> general
> > > >> >>>>>>>>>> executors of "sync" and "async" tasks respectively.
> That's
> > > >> >> where
> > > >> >>>>>> Airflow
> > > >> >>>>>>>>>> is
> > > >> >>>>>>>>>> evolving towards - inevitably.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> But we can of course phase things in out for
> > implementation -
> > > >> >>> even
> > > >> >>>>> if
> > > >> >>>>>> AIP
> > > >> >>>>>>>>>> should cover both, I think if the goal of the AIP and
> > > preamble
> > > >> >>> is
> > > >> >>>>>> about
> > > >> >>>>>>>>>> separating "user code" from "database" as the main
> reason,
> > it
> > > >> >>> also
> > > >> >>>>>> means
> > > >> >>>>>>>>>> Triggerer if you ask me (from design point of view at
> > least).
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> Again implementation can be phased and even different
> > people
> > > >> >> and
> > > >> >>>>> teams
> > > >> >>>>>>>>>> might work on those phases/pieces.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> J.
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > >> >>>>>> sumeet.manit@gmail.com
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>> wrote:
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> triggerers
> > > as
> > > >> >>>> well,
> > > >> >>>>>>>>>> but
> > > >> >>>>>>>>>>>> that
> > > >> >>>>>>>>>>>> can be done as part of a different AIP
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> You won't achieve your goal of "true" isolation of user
> > code
> > > >> >> if
> > > >> >>>> you
> > > >> >>>>>>>>>> don't
> > > >> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it - it
> > > >> >> should
> > > >> >>>>> cover
> > > >> >>>>>>>>>>> both.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> My bad, should've explained our architecture for
> triggers
> > as
> > > >> >>>> well,
> > > >> >>>>>>>>>>> apologies. So here it is:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> - Triggers would be running on a centralized service,
> so
> > > >> >> all
> > > >> >>>> the
> > > >> >>>>>>>>>> Trigger
> > > >> >>>>>>>>>>> classes will be part of the platform team's repo and
> not
> > > >> >> the
> > > >> >>>>>>>>>> customer's
> > > >> >>>>>>>>>>> repo
> > > >> >>>>>>>>>>> - The triggers won't be able to use any libs other than
> > std
> > > >> >>>> ones,
> > > >> >>>>>>>>>> which
> > > >> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> > > >> >>>>>>>>>>> - As we are the owners of the core Airflow repo,
> > customers
> > > >> >>> have
> > > >> >>>>> to
> > > >> >>>>>>>>>> get
> > > >> >>>>>>>>>>> our approval to land any class in this path (unlike the
> > > >> >> dags
> > > >> >>>> repo
> > > >> >>>>>>>>>> which
> > > >> >>>>>>>>>>> they own)
> > > >> >>>>>>>>>>> - When a customer's task defer, we would have an
> > allowlist
> > > >> >> on
> > > >> >>>> our
> > > >> >>>>>>>>>> side
> > > >> >>>>>>>>>>> to check if we should do the async polling or not
> > > >> >>>>>>>>>>> - If the Trigger class isn't part of our repo
> > (allowlist),
> > > >> >>> just
> > > >> >>>>>>>>>> fail the
> > > >> >>>>>>>>>>> task, as anyway we won't be having the code that they
> > used
> > > >> >> in
> > > >> >>>> the
> > > >> >>>>>>>>>>> trigger
> > > >> >>>>>>>>>>> class
> > > >> >>>>>>>>>>> - If any of these conditions aren't suitable for you
> (as
> > a
> > > >> >>>>>>>>>> customer),
> > > >> >>>>>>>>>>> feel free to use sync tasks only
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> > > >> >> communicate
> > > >> >>>> over
> > > >> >>>>>>>>>> apis
> > > >> >>>>>>>>>>> only. If that is done, then we can have instances of
> > > >> >> triggerer
> > > >> >>>> svc
> > > >> >>>>>>>>>> running
> > > >> >>>>>>>>>>> at customer's side as well, which can process any type
> of
> > > >> >>> trigger
> > > >> >>>>>>>>>> class.
> > > >> >>>>>>>>>>> Though that's not a blocker for us at the moment, cause
> > > >> >>> triggerer
> > > >> >>>>> are
> > > >> >>>>>>>>>>> mostly doing just polling using simple libs like
> requests.
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > >> >>>>>>>>>> <ikholopov@google.com.invalid
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I
> believe
> > > it
> > > >> >>>>> aligns
> > > >> >>>>>>>>>> well
> > > >> >>>>>>>>>>>> with the goals of making Airflow well-scalable beyond a
> > > >> >>>>> single-team
> > > >> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full
> > support
> > > >> >>> with
> > > >> >>>>> this
> > > >> >>>>>>>>>> one.
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> I've left a couple of clarification requests on the AIP
> > > >> >> page.
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> Thanks,
> > > >> >>>>>>>>>>>> Igor
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > >> >>>>>>>>>>> sumeet.manit@gmail.com>
> > > >> >>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's
> good
> > to
> > > >> >>> know
> > > >> >>>>>>>>>> that
> > > >> >>>>>>>>>>> the
> > > >> >>>>>>>>>>>>> DAG processor has some preemptive measures in place to
> > > >> >>> prevent
> > > >> >>>>>>>>>> access
> > > >> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to
> > solve
> > > >> >> is
> > > >> >>>> not
> > > >> >>>>> to
> > > >> >>>>>>>>>>>> provide
> > > >> >>>>>>>>>>>>> DB creds to the customer teams, who are using Airflow
> > as a
> > > >> >>>>>>>>>> multi-tenant
> > > >> >>>>>>>>>>>>> orchestration platform. I've updated the doc to
> reflect
> > > >> >> this
> > > >> >>>>> point
> > > >> >>>>>>>>>> as
> > > >> >>>>>>>>>>>> well.
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> Answering Jarek's points,
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism,
> > added
> > > >> >>> that
> > > >> >>>>> in
> > > >> >>>>>>>>>> doc,
> > > >> >>>>>>>>>>>> but
> > > >> >>>>>>>>>>>>> still how the token can be obtained (safely) is still
> > open
> > > >> >> in
> > > >> >>>> my
> > > >> >>>>>>>>>> mind.
> > > >> >>>>>>>>>>> I
> > > >> >>>>>>>>>>>>> believe the token used by task executors can be
> created
> > > >> >>> outside
> > > >> >>>>> of
> > > >> >>>>>>>>>> it
> > > >> >>>>>>>>>>> as
> > > >> >>>>>>>>>>>>> well (I may be wrong here).
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for
> triggerers
> > > as
> > > >> >>>> well,
> > > >> >>>>>>>>>> but
> > > >> >>>>>>>>>>>> that
> > > >> >>>>>>>>>>>>> can be done as part of a different AIP
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work largely.
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we
> > can
> > > >> >>> work
> > > >> >>>>>>>>>> with
> > > >> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would be
> > > >> >> treated
> > > >> >>>> as
> > > >> >>>>> a
> > > >> >>>>>>>>>> diff
> > > >> >>>>>>>>>>>>> bundle.
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> Also, added points around callbacks, as these are also
> > > >> >>> fetched
> > > >> >>>>>>>>>> directly
> > > >> >>>>>>>>>>>>> from the DB.
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > > >> >>>> jarek@potiuk.com>
> > > >> >>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> > likely
> > > >> >>> not
> > > >> >>>>>>>>>>>> protection
> > > >> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> > > >> >>> protect
> > > >> >>>>>>>>>> against
> > > >> >>>>>>>>>>>>>> casual DB access attempts - the db session is blanked
> > out
> > > >> >> in
> > > >> >>>> the
> > > >> >>>>>>>>>>>> parsing
> > > >> >>>>>>>>>>>>>> process , as are the env var configs
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > >> >>>>>>>>>>>>>> -
> > > >> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no
> > protection
> > > >> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed
> back
> > > >> >> then
> > > >> >>> in
> > > >> >>>>>>>>>> March
> > > >> >>>>>>>>>>> I
> > > >> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with
> full
> > > >> >>>> knowledge
> > > >> >>>>>>>>>> it's
> > > >> >>>>>>>>>>>> not
> > > >> >>>>>>>>>>>>>> protecting against all threats.
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > > >> >>>>>>>>>> ash@apache.org>
> > > >> >>>>>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> > likely
> > > >> >>> not
> > > >> >>>>>>>>>>>> protection
> > > >> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it
> does
> > > >> >>> protect
> > > >> >>>>>>>>>>> against
> > > >> >>>>>>>>>>>>>>> casual DB access attempts - the db session is
> blanked
> > > out
> > > >> >>> in
> > > >> >>>>>>>>>> the
> > > >> >>>>>>>>>>>>> parsing
> > > >> >>>>>>>>>>>>>>> process , as are the env var configs
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > >> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> > > >> >> protection
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> > > >> >> jarek@potiuk.com>
> > > >> >>>>>>>>>> wrote:
> > > >> >>>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no
> > > built-in
> > > >> >>>>>>>>>>> protection
> > > >> >>>>>>>>>>>>>>> against
> > > >> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example - read
> > > >> >>> database
> > > >> >>>>>>>>>>>>>>>> credentials from airflow configuration and use them
> > to
> > > >> >>> talk
> > > >> >>>>>>>>>> to DB
> > > >> >>>>>>>>>>>>>>> directly.
> > > >> >>>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>>
> > > >> >>>>>>>>>>>>
> > > >> >>>>>>>>>>>
> > > >> >>>>>>>>>>
> > > >> >>>>>>>>>
> > > >> >>>>>>
> > > >> >>>>>>
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >>
> > > >>
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
Will do. We are also discussing - for now - within the security team - the
various aspects of authentication approach we want to have - for both
"UI/User authentication" as well as "Long running services" and how they
relate to token exchange, invalidation and other scenarios. What we will
come up - I hope shortly - is a proposal of general "model" of all kinds of
security and authentication scenarios, so that we do not have to reinvent
the wheel and try to figure out all the aspects with individual AIPs. We
are not far from bringing it to the open discussion at devlist, but we have
to be careful about some of the aspects that we might need to improve in
the current setup to close some - small - loopholes so bear with us :)
On Wed, Aug 13, 2025 at 9:33 AM Sumit Maheshwari <su...@gmail.com>
wrote:
> Thanks Ash and Jarek, for the detailed comments. I largely agree with the
> points mentioned by you guys, hence I updated the AIP and added a section
> on Authentication, API Versioning, and Packaging as well. Please go through
> it once more and let me know if there are more things to consider before I
> open it for voting.
>
>
> On Thu, Aug 7, 2025 at 6:03 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Also
> >
> > > 1. The Authentication token. How will this long lived token work
> without
> > being insecure. Who and what will generate it? How will we identify
> > top-level requests for Variables in order to be able to add Variable
> > RBAC/ACLs. This is an important enough thing that I think it needs
> > discussion before we vote on this AIP.
> >
> > We are currently discussing - in the security team - approach for JWT
> token
> > handling, so likely we could move the discussion there, it does have some
> > security implications and I think we should bring our finding to the
> > devlist when we complete it, but I think we should add this case there.
> > IMHO we should have a different approach for UI, different for Tasks,
> > different for Triggerer, and different for DagProcessor. (possibly the
> > Trigerer and DagProcessor could be the same because they share
> essentially
> > the same long-living token. Ash - I will add this to the discussion
> there.
> >
> > J.
> >
> >
> >
> > On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Ah.. So if we are talking about a more complete approach - seeing those
> > > comments from Ash - make me think if we should have another AIP.
> > > (connected) about splitting the distributions. We have never finalized
> it
> > > (nor even discussed id) but Ash - you had some initial document for
> that.
> > > So maybe we should finalize it and rather than specify it in this AIP -
> > > have a separate AIP about distribution split that AIP-92 could depend
> on.
> > > It seems much more reasonable to split "distribution and code split"
> from
> > > parsing isolation I think and implement them separately/in parallel.
> > >
> > > Reading Ash comments (and maybe I am going a bit further than Ash) it
> > > calls for something that I am a big proponent of - splitting
> > "airflow-core"
> > > and having a different "scheduler". "webserver", "dag processor" and
> > > "triggerer" distributions. Now - we have the capability of having
> > "shared"
> > > code - we do not need "common" code to make it happen - because we can
> > > share code.
> > >
> > > What it could give us - on top of clean client/server split, we could
> > have
> > > different dependencies used by those distributions. Additionally, we
> > could
> > > also split-off the executors from providers and finally implement it in
> > the
> > > way that scheduler does not use providers at all (not even
> > cncf.kubernetes
> > > nor celery providers installed in scheduler nor webserver but
> "executors"
> > > packages instead. The code sharing approach with symlinks we have now
> > will
> > > make it a .... breeze :) . That would also imply sharing "connection"
> > > definitions through DB, and likely implementing "test connection"
> > > feature properly finally (i.e executing test connection in worker /
> > > triggerer rather than in web server which is a reason why we disabled
> it
> > by
> > > default now). This way "api-server" would not need any of the providers
> > to
> > > be installed either which IMHO is the biggest win from a security point
> > of
> > > view.
> > >
> > > And the nice thing about it is that it would be rather transparent when
> > > anyone uses "pip install apache-airflow" - it would behave exactly the
> > > same, no more complexity involved, simply more distributions installed
> > when
> > > 'apache-airflow" meta-distribution is used, but it would allow those
> who
> > > want to implement a more complex and secure setup to have different
> > > "environments" with modularized pieces of airflow installed - only
> > > "apache-airflow-dag-processor + task-sdk + providers" where
> dag-processor
> > > is run, only "apache-airflow-scheduler + executors" where scheduler is
> > > installed only "apache-airflow-task-sdk + providers" where workers are
> > > running, only "apache-airflow-api-server" where api-server is running
> and
> > > only "apache-airflow-trigger + task-sdk + providers" .
> > >
> > > I am happy (Ash If you are fine with that) to take that original
> document
> > > over and lead this part and new AIP to completion (including
> > > implementation), I am very much convinced that this will lead to much
> > > better dependency security and more modular code without impacting the
> > > "apache-airflow" installation complexity.
> > >
> > > If we do it this way- the part of code/clean split would be "delegated
> > > out" from AIP-92 to this new AIP and turned into dependency.
> > >
> > > J.
> > >
> > >
> > > On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >
> > >> This AIP is definitely heading in the right direction and is a feature
> > >> I’d like to see.
> > >>
> > >> For me the outstanding things that need more detail:
> > >>
> > >> 1. The Authentication token. How will this long lived token work
> without
> > >> being insecure. Who and what will generate it? How will we identify
> > >> top-level requests for Variables in order to be able to add Variable
> > >> RBAC/ACLs. This is an important enough thing that I think it needs
> > >> discussion before we vote on this AIP.
> > >> 2. Security generally — how will this work, especially with the
> > >> multi-team? I think this likely means making the APIs work on the
> bundle
> > >> level as you mention in the doc, but I haven’t thought deeply about
> this
> > >> yet.
> > >> 3. API Versioning? One of the the key driving goals with AIP-72 and
> the
> > >> Task Execution SDK was the idea that “you can upgrade the API server
> as
> > you
> > >> like, and your clients/workers never need to work” — i.e. the API
> > server is
> > >> 100% working with all older versions of the TaskSDK. I don’t know if
> we
> > >> will achieve that goal in the long run but it is the desire, and part
> of
> > >> why we are using CalVer and the Cadwyn library to provide API
> > versioning.
> > >> 4. As mentioned previously, not sure the existing serialised JSON
> format
> > >> for DAGs is correct, but since that now has version and we already
> have
> > the
> > >> ability to upgrade that somewhere in the Airflow Core that doesn’t
> > >> necessarily become a blocker/pre-requisite for this AIP.
> > >>
> > >> I think Dag parsing API client+submission+parsing process manager
> should
> > >> either live in the Task SDK dist, or in a new separate dist that uses
> > >> TaskSDK, but crucially not in apache-airflow-core. My reason for this
> is
> > >> that I want it to be possible for the server components (scheduler,
> API
> > >> server) to not need task-sdk installed (just for cleanliness/avoiding
> > >> confusion about what versions it needs) and also vice-verse, to be
> able
> > to
> > >> run a “team worker bundle” (Dag parsing, workers, triggered/async
> > workers)
> > >> on whatever version of TaskSDK they choose, again without
> > >> apache-airflow-core installed for avoidance of doubt.
> > >>
> > >> Generally I would like this as it means we can have a nicer separation
> > of
> > >> Core and Dag parsing code, as the dag parsing itself uses the SDK, it
> > would
> > >> be nice to have a proper server/client split, both from a tighter
> > security
> > >> point-of-view, but also from a code layout point of view.
> > >>
> > >> -ash
> > >>
> > >>
> > >> > On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >> >
> > >> > Well, you started it - so it's up to you to decide if you think we
> > have
> > >> > consensus, or whether we need a vote.
> > >> >
> > >> > And It's not a question of "informal" vote but it's rather clear
> > >> following
> > >> > the https://www.apache.org/foundation/voting.html that we either
> > need a
> > >> > LAZY CONSENSUS or VOTE thread. Both are formal.
> > >> >
> > >> > This is the difficult part when you have a proposal, to assess (by
> > you)
> > >> > whether we are converging to consensus or whether vote is needed.
> > There
> > >> is
> > >> > no other body or "authority" to do it for you.
> > >> >
> > >> > J.
> > >> >
> > >> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <
> > sumeet.manit@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> >> Sorry for nudging again, but can we get into some consensus on
> this?
> > I
> > >> mean
> > >> >> if this AIP isn't good enough, then we can drop it altogether and
> > >> someone
> > >> >> can rethink the whole thing. Should we do some kind of informal
> > voting
> > >> and
> > >> >> close this thread?
> > >> >>
> > >> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > >> >>
> > >> >>>>> My main concern with this right now is the serialisation format
> of
> > >> >> DAGs
> > >> >>> —
> > >> >>>> it wasn’t really designed with remote submission in mind, so it
> > need
> > >> >> some
> > >> >>>> careful examination to see if it is fit for this purpose or not.
> > >> >>>
> > >> >>> I understand Ash's concerns - the format has not been designed
> with
> > >> >>> size/speed optimization in mind so **possibly** we could design a
> > >> >> different
> > >> >>> format that would be better suited.
> > >> >>>
> > >> >>> BUT ... Done is better than perfect.
> > >> >>>
> > >> >>> I think there are a number of risks involved in changing the
> format
> > >> and
> > >> >> it
> > >> >>> could significantly increase time of development with uncertain
> > gains
> > >> at
> > >> >>> the end - also because of the progress in compression that
> happened
> > >> over
> > >> >>> the last few years.
> > >> >>>
> > >> >>> It might be a good idea to experiment a bit with different
> > compression
> > >> >>> algorithms for "our" dag representation and possibly we could find
> > the
> > >> >> best
> > >> >>> algorithm for "airflow dag" type of json data. There are a lot of
> > >> >>> repetitions in the JSON representation and I guess in "our" json
> > >> >>> representation there are some artifacts and repeated sections that
> > >> simply
> > >> >>> might compress well with different algorithms. Also in this case
> > >> >>> speed matters (and CPU trade-off).
> > >> >>>
> > >> >>> Looking at compression "theory" - before we experiment with it -
> > >> there is
> > >> >>> the relatively new standard "zstandard"
> > >> https://github.com/facebook/zstd
> > >> >>> compression opensourced in 2016 which I've heard good things
> about -
> > >> >>> especially that it maintains a very good compression rate for text
> > >> data,
> > >> >>> but also it is tremendously fast - especially for decompression
> > >> (which is
> > >> >>> super important factor for us - we compress new DAG representation
> > far
> > >> >> less
> > >> >>> often than decompress it in general case). It is standardized in
> RFC
> > >> >>> https://datatracker.ietf.org/doc/html/rfc8878 and there are
> various
> > >> >>> implementations and it is even being added to Python standard
> > library
> > >> in
> > >> >>> Python 3.14
> > >> https://docs.python.org/3.14/library/compression.zstd.html
> > >> >> and
> > >> >>> there is a very well maintained python binding library
> > >> >>> https://pypi.org/project/zstd/ to Yann Collet (algorithm author)
> > >> ZSTD C
> > >> >>> library. And libzstd is already part of our images - it is needed
> by
> > >> >> other
> > >> >>> dependencies of ours. All with BSD licence, directly usable by us.
> > >> >>>
> > >> >>> I think this one might be a good candidate for us to try, and
> > possibly
> > >> >> with
> > >> >>> zstd we could achieve both size and CPU overhead that would be
> > >> comparable
> > >> >>> with any "new" format we could come up with - especially that we
> are
> > >> >>> talking merely about processing a huge blob between "storable"
> > >> >> (compressed)
> > >> >>> and "locally usable" state (Python dict). We could likely use a
> > >> streaming
> > >> >>> JSON library (say the one that is used in Pydantic internally
> > >> >>> https://github.com/pydantic/jiter - we already have it as part of
> > >> >>> Pydantic)
> > >> >>> to also save memory - we could stream decompressed stream into
> > jitter
> > >> so
> > >> >>> that both the json dict and string representation does not have to
> > be
> > >> >>> loaded fully in memory at the same time. There are likely lots of
> > >> >>> optimisations we could do - I mentioned possibly streaming the
> data
> > >> from
> > >> >>> API directly to DB (if this is possible - not sure)
> > >> >>>
> > >> >>> J.
> > >> >>>
> > >> >>>
> > >> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> > >> sumeet.manit@gmail.com>
> > >> >>> wrote:
> > >> >>>
> > >> >>>>>
> > >> >>>>> My main concern with this right now is the serialisation format
> of
> > >> >>> DAGs —
> > >> >>>>> it wasn’t really designed with remote submission in mind, so it
> > need
> > >> >>> some
> > >> >>>>> careful examination to see if it is fit for this purpose or not.
> > >> >>>>>
> > >> >>>>
> > >> >>>> I'm not sure on this point, cause if we are able to convert a DAG
> > >> into
> > >> >>>> JSON, then it has to be transferable over the internet.
> > >> >>>>
> > >> >>>> In particular One of the things I worry about is that the JSON
> can
> > >> get
> > >> >>> huge
> > >> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> > >> >>>>
> > >> >>>>
> > >> >>>> Yeah, agree on this, thats why we can transfer compressed data
> > >> instead
> > >> >> of
> > >> >>>> real json. Of course, this won't guarantee that the payload will
> > >> always
> > >> >>> be
> > >> >>>> small enough, but we can't say that it'll definitely happen
> either.
> > >> >>>>
> > >> >>>> I also wonder if as part of this proposal we should move the
> > Callback
> > >> >>>>> requests off the dag parsers and on to the workers instead
> > >> >>>>
> > >> >>>> let's make such a "workfload" implementation stream that could
> > >> support
> > >> >>> both
> > >> >>>>> - Deadlines and DAG parsing logic
> > >> >>>>
> > >> >>>>
> > >> >>>> I don't have any strong opinion here, but it feels like it's
> gonna
> > >> blow
> > >> >>> up
> > >> >>>> the scope of the AIP too much.
> > >> >>>>
> > >> >>>>
> > >> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com>
> > >> wrote:
> > >> >>>>
> > >> >>>>>> My main concern with this right now is the serialisation format
> > of
> > >> >>>> DAGs —
> > >> >>>>> it wasn’t really designed with remote submission in mind, so it
> > need
> > >> >>> some
> > >> >>>>> careful examination to see if it is fit for this purpose or not.
> > >> >>>>>
> > >> >>>>> Yep. That might be potentially a problem (or at least "need more
> > >> >>>> resources
> > >> >>>>> to run airflow") and that is where my "2x memory" came from if
> we
> > do
> > >> >> it
> > >> >>>> in
> > >> >>>>> a trivial way. Currently we a) keep the whole DAG in memory when
> > >> >>>>> serializing it b) submit it to database (also using essentially
> > some
> > >> >>> kind
> > >> >>>>> of API (implemented by the database client) - so we know the
> whole
> > >> >>> thing
> > >> >>>>> "might work" but indeed if you use a trivial implementation of
> > >> >>> submitting
> > >> >>>>> the whole json - it basically means that the whole json will
> have
> > to
> > >> >>> also
> > >> >>>>> be kept in the memory of API server. But we also compress it
> when
> > >> >>> needed
> > >> >>>> -
> > >> >>>>> I wonder what are the compression ratios we saw with those
> > 10-20MBs
> > >> >>> Dags
> > >> >>>> -
> > >> >>>>> if the problem is using strings where bool would suffice,
> > >> compression
> > >> >>>>> should generally help a lot. We could only ever send compressed
> > data
> > >> >>> over
> > >> >>>>> the API - there seems to be no need to send "plain JSON" data
> over
> > >> >> the
> > >> >>>> API
> > >> >>>>> or storing the plain JSON in the DB (of course that trades
> memory
> > >> for
> > >> >>>> CPU).
> > >> >>>>>
> > >> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
> > >> >> support
> > >> >>>> for
> > >> >>>>> any kind if binary data streaming - because that could help a
> lot
> > of
> > >> >> if
> > >> >>>> we
> > >> >>>>> could use streaming HTTP API and chunk and append the binary
> > chunks
> > >> >>> (when
> > >> >>>>> writing) - or read data in chunks ans stream them back via the
> > API.
> > >> >>> That
> > >> >>>>> could seriously decrease the amount of memory needed by the API
> > >> >> server
> > >> >>> to
> > >> >>>>> process such huge serialized dags.
> > >> >>>>>
> > >> >>>>> And yeah - I would also love the "execute task" to be
> implemented
> > >> >> here
> > >> >>> -
> > >> >>>>> but I am not sure if this should be part of the same effort or
> > maybe
> > >> >> a
> > >> >>>>> separate implementation? That sounds very loosely coupled with
> DB
> > >> >>>>> isolation. And it seems a common theme - I think that would also
> > >> make
> > >> >>> the
> > >> >>>>> sync Deadline alerts case that we discussed at the dev call
> > today. I
> > >> >>>> wonder
> > >> >>>>> if that should not be kind of parallel (let's make such a
> > >> "workfload"
> > >> >>>>> implementation stream that could support both - Deadlines and
> DAG
> > >> >>> parsing
> > >> >>>>> logic. We have already two "users" for it and I really love the
> > >> >> saying
> > >> >>>> "if
> > >> >>>>> you want to make something reusable - make it usable first" -
> > seems
> > >> >>> like
> > >> >>>>> we might have good opportunity to make such workload
> > implementation
> > >> >>>> "doubly
> > >> >>>>> used" from the beginning which would increase chances it will
> be
> > >> >>>>> "reusable" for other things as well :).
> > >> >>>>>
> > >> >>>>> J.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <
> > ash@apache.org>
> > >> >>>> wrote:
> > >> >>>>>
> > >> >>>>>> My main concern with this right now is the serialisation format
> > of
> > >> >>>> DAGs —
> > >> >>>>>> it wasn’t really designed with remote submission in mind, so it
> > >> >> need
> > >> >>>> some
> > >> >>>>>> careful examination to see if it is fit for this purpose or
> not.
> > >> >>>>>>
> > >> >>>>>> In particular One of the things I worry about is that the JSON
> > can
> > >> >>> get
> > >> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!)
> > (which
> > >> >> is
> > >> >>>>>> likely due to things being included as text when a bool might
> > >> >>> suffice,
> > >> >>>>> for
> > >> >>>>>> example) But I don’t think “just submit the existing JSON over
> an
> > >> >>> API”
> > >> >>>>> is a
> > >> >>>>>> good idea.
> > >> >>>>>>
> > >> >>>>>> I also wonder if as part of this proposal we should move the
> > >> >> Callback
> > >> >>>>>> requests off the dag parsers and on to the workers instead — in
> > >> >>> AIP-72
> > >> >>>> we
> > >> >>>>>> introduced the concept of a Workload, with the only one
> existing
> > >> >>> right
> > >> >>>>> now
> > >> >>>>>> is “ExecuteTask”
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > >> >>>>>> — it might be time to finally move task and dag callbacks to
> the
> > >> >> same
> > >> >>>>> thing
> > >> >>>>>> and make dag parsers only responsible for, well, parsing. :)
> > >> >>>>>>
> > >> >>>>>> These are all solvable problems, and this will be a great
> feature
> > >> >> to
> > >> >>>>> have,
> > >> >>>>>> but we need to do some more thinking and planning first.
> > >> >>>>>>
> > >> >>>>>> -ash
> > >> >>>>>>
> > >> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> > >> >> sumeet.manit@gmail.com
> > >> >>>>
> > >> >>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>> Gentle reminder for everyone to review the proposal.
> > >> >>>>>>>
> > >> >>>>>>> Updated link:
> > >> >>>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > >> >>>>>>>
> > >> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > >> >>>>> sumeet.manit@gmail.com
> > >> >>>>>>>
> > >> >>>>>>> wrote:
> > >> >>>>>>>
> > >> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and others
> > >> >>>>> suggested, I
> > >> >>>>>>>> expanded the scope of this AIP and divided it into three
> > phases.
> > >> >>>> With
> > >> >>>>>> the
> > >> >>>>>>>> increased scope, the boundary line between this AIP and
> AIP-85
> > >> >>> got a
> > >> >>>>>> little
> > >> >>>>>>>> thinner, but I believe these are still two different
> > >> >> enhancements
> > >> >>> to
> > >> >>>>>> make.
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>>
> > >> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > >> >>>>>> sumeet.manit@gmail.com>
> > >> >>>>>>>> wrote:
> > >> >>>>>>>>
> > >> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well to
> be
> > >> >>> part
> > >> >>>>> of
> > >> >>>>>>>>> this AIP and phase out the implementation. Though I didn't
> > >> >>> exclude
> > >> >>>>>> Triggers
> > >> >>>>>>>>> because "Uber" doesn't need that, I just thought of keeping
> > the
> > >> >>>> scope
> > >> >>>>>> of
> > >> >>>>>>>>> development small and achieving them, just like it was done
> in
> > >> >>>>> Airlfow
> > >> >>>>>> 3 by
> > >> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
> > >> >>>>>>>>>
> > >> >>>>>>>>> But if you think Triggers should be part of this AIP itself,
> > >> >>> then I
> > >> >>>>> can
> > >> >>>>>>>>> do that and include Triggers as well in it.
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <
> > jarek@potiuk.com
> > >> >>>
> > >> >>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>>> I would very much prefer the architectural choices of this
> > AIP
> > >> >>> are
> > >> >>>>>> based
> > >> >>>>>>>>>> on
> > >> >>>>>>>>>> "general public" needs rather than "Uber needs" even if
> Uber
> > >> >>> will
> > >> >>>> be
> > >> >>>>>>>>>> implementing it - so from my point of view having Trigger
> > >> >>>> separation
> > >> >>>>>> as
> > >> >>>>>>>>>> part of it is quite important.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> But that's not even this.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> We've been discussing for example for Deadlines (being
> > >> >>> implemented
> > >> >>>>> by
> > >> >>>>>>>>>> Dennis and Ramit a possibility of short,
> notification-style
> > >> >>>>>> "deadlines"
> > >> >>>>>>>>>> to be send to triggerer for execution - this is well
> advanced
> > >> >>> now,
> > >> >>>>> and
> > >> >>>>>>>>>> whether you want it or not Dag-provided code might be
> > >> >> serialized
> > >> >>>> and
> > >> >>>>>> sent
> > >> >>>>>>>>>> to triggerer for execution. This is part of our "broader"
> > >> >>>>>> architectural
> > >> >>>>>>>>>> change where we treat "workers" and "triggerer" similarly
> as
> > a
> > >> >>>>> general
> > >> >>>>>>>>>> executors of "sync" and "async" tasks respectively. That's
> > >> >> where
> > >> >>>>>> Airflow
> > >> >>>>>>>>>> is
> > >> >>>>>>>>>> evolving towards - inevitably.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> But we can of course phase things in out for
> implementation -
> > >> >>> even
> > >> >>>>> if
> > >> >>>>>> AIP
> > >> >>>>>>>>>> should cover both, I think if the goal of the AIP and
> > preamble
> > >> >>> is
> > >> >>>>>> about
> > >> >>>>>>>>>> separating "user code" from "database" as the main reason,
> it
> > >> >>> also
> > >> >>>>>> means
> > >> >>>>>>>>>> Triggerer if you ask me (from design point of view at
> least).
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Again implementation can be phased and even different
> people
> > >> >> and
> > >> >>>>> teams
> > >> >>>>>>>>>> might work on those phases/pieces.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> J.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > >> >>>>>> sumeet.manit@gmail.com
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>> wrote:
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers
> > as
> > >> >>>> well,
> > >> >>>>>>>>>> but
> > >> >>>>>>>>>>>> that
> > >> >>>>>>>>>>>> can be done as part of a different AIP
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> You won't achieve your goal of "true" isolation of user
> code
> > >> >> if
> > >> >>>> you
> > >> >>>>>>>>>> don't
> > >> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it - it
> > >> >> should
> > >> >>>>> cover
> > >> >>>>>>>>>>> both.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> My bad, should've explained our architecture for triggers
> as
> > >> >>>> well,
> > >> >>>>>>>>>>> apologies. So here it is:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> - Triggers would be running on a centralized service, so
> > >> >> all
> > >> >>>> the
> > >> >>>>>>>>>> Trigger
> > >> >>>>>>>>>>> classes will be part of the platform team's repo and not
> > >> >> the
> > >> >>>>>>>>>> customer's
> > >> >>>>>>>>>>> repo
> > >> >>>>>>>>>>> - The triggers won't be able to use any libs other than
> std
> > >> >>>> ones,
> > >> >>>>>>>>>> which
> > >> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> > >> >>>>>>>>>>> - As we are the owners of the core Airflow repo,
> customers
> > >> >>> have
> > >> >>>>> to
> > >> >>>>>>>>>> get
> > >> >>>>>>>>>>> our approval to land any class in this path (unlike the
> > >> >> dags
> > >> >>>> repo
> > >> >>>>>>>>>> which
> > >> >>>>>>>>>>> they own)
> > >> >>>>>>>>>>> - When a customer's task defer, we would have an
> allowlist
> > >> >> on
> > >> >>>> our
> > >> >>>>>>>>>> side
> > >> >>>>>>>>>>> to check if we should do the async polling or not
> > >> >>>>>>>>>>> - If the Trigger class isn't part of our repo
> (allowlist),
> > >> >>> just
> > >> >>>>>>>>>> fail the
> > >> >>>>>>>>>>> task, as anyway we won't be having the code that they
> used
> > >> >> in
> > >> >>>> the
> > >> >>>>>>>>>>> trigger
> > >> >>>>>>>>>>> class
> > >> >>>>>>>>>>> - If any of these conditions aren't suitable for you (as
> a
> > >> >>>>>>>>>> customer),
> > >> >>>>>>>>>>> feel free to use sync tasks only
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> > >> >> communicate
> > >> >>>> over
> > >> >>>>>>>>>> apis
> > >> >>>>>>>>>>> only. If that is done, then we can have instances of
> > >> >> triggerer
> > >> >>>> svc
> > >> >>>>>>>>>> running
> > >> >>>>>>>>>>> at customer's side as well, which can process any type of
> > >> >>> trigger
> > >> >>>>>>>>>> class.
> > >> >>>>>>>>>>> Though that's not a blocker for us at the moment, cause
> > >> >>> triggerer
> > >> >>>>> are
> > >> >>>>>>>>>>> mostly doing just polling using simple libs like requests.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > >> >>>>>>>>>> <ikholopov@google.com.invalid
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I believe
> > it
> > >> >>>>> aligns
> > >> >>>>>>>>>> well
> > >> >>>>>>>>>>>> with the goals of making Airflow well-scalable beyond a
> > >> >>>>> single-team
> > >> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full
> support
> > >> >>> with
> > >> >>>>> this
> > >> >>>>>>>>>> one.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> I've left a couple of clarification requests on the AIP
> > >> >> page.
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> Thanks,
> > >> >>>>>>>>>>>> Igor
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > >> >>>>>>>>>>> sumeet.manit@gmail.com>
> > >> >>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's good
> to
> > >> >>> know
> > >> >>>>>>>>>> that
> > >> >>>>>>>>>>> the
> > >> >>>>>>>>>>>>> DAG processor has some preemptive measures in place to
> > >> >>> prevent
> > >> >>>>>>>>>> access
> > >> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to
> solve
> > >> >> is
> > >> >>>> not
> > >> >>>>> to
> > >> >>>>>>>>>>>> provide
> > >> >>>>>>>>>>>>> DB creds to the customer teams, who are using Airflow
> as a
> > >> >>>>>>>>>> multi-tenant
> > >> >>>>>>>>>>>>> orchestration platform. I've updated the doc to reflect
> > >> >> this
> > >> >>>>> point
> > >> >>>>>>>>>> as
> > >> >>>>>>>>>>>> well.
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> Answering Jarek's points,
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism,
> added
> > >> >>> that
> > >> >>>>> in
> > >> >>>>>>>>>> doc,
> > >> >>>>>>>>>>>> but
> > >> >>>>>>>>>>>>> still how the token can be obtained (safely) is still
> open
> > >> >> in
> > >> >>>> my
> > >> >>>>>>>>>> mind.
> > >> >>>>>>>>>>> I
> > >> >>>>>>>>>>>>> believe the token used by task executors can be created
> > >> >>> outside
> > >> >>>>> of
> > >> >>>>>>>>>> it
> > >> >>>>>>>>>>> as
> > >> >>>>>>>>>>>>> well (I may be wrong here).
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers
> > as
> > >> >>>> well,
> > >> >>>>>>>>>> but
> > >> >>>>>>>>>>>> that
> > >> >>>>>>>>>>>>> can be done as part of a different AIP
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work largely.
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we
> can
> > >> >>> work
> > >> >>>>>>>>>> with
> > >> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would be
> > >> >> treated
> > >> >>>> as
> > >> >>>>> a
> > >> >>>>>>>>>> diff
> > >> >>>>>>>>>>>>> bundle.
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> Also, added points around callbacks, as these are also
> > >> >>> fetched
> > >> >>>>>>>>>> directly
> > >> >>>>>>>>>>>>> from the DB.
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > >> >>>> jarek@potiuk.com>
> > >> >>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> likely
> > >> >>> not
> > >> >>>>>>>>>>>> protection
> > >> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> > >> >>> protect
> > >> >>>>>>>>>> against
> > >> >>>>>>>>>>>>>> casual DB access attempts - the db session is blanked
> out
> > >> >> in
> > >> >>>> the
> > >> >>>>>>>>>>>> parsing
> > >> >>>>>>>>>>>>>> process , as are the env var configs
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > >> >>>>>>>>>>>>>> -
> > >> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no
> protection
> > >> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed back
> > >> >> then
> > >> >>> in
> > >> >>>>>>>>>> March
> > >> >>>>>>>>>>> I
> > >> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with full
> > >> >>>> knowledge
> > >> >>>>>>>>>> it's
> > >> >>>>>>>>>>>> not
> > >> >>>>>>>>>>>>>> protecting against all threats.
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > >> >>>>>>>>>> ash@apache.org>
> > >> >>>>>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is
> likely
> > >> >>> not
> > >> >>>>>>>>>>>> protection
> > >> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> > >> >>> protect
> > >> >>>>>>>>>>> against
> > >> >>>>>>>>>>>>>>> casual DB access attempts - the db session is blanked
> > out
> > >> >>> in
> > >> >>>>>>>>>> the
> > >> >>>>>>>>>>>>> parsing
> > >> >>>>>>>>>>>>>>> process , as are the env var configs
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > >> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> > >> >> protection
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> > >> >> jarek@potiuk.com>
> > >> >>>>>>>>>> wrote:
> > >> >>>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no
> > built-in
> > >> >>>>>>>>>>> protection
> > >> >>>>>>>>>>>>>>> against
> > >> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example - read
> > >> >>> database
> > >> >>>>>>>>>>>>>>>> credentials from airflow configuration and use them
> to
> > >> >>> talk
> > >> >>>>>>>>>> to DB
> > >> >>>>>>>>>>>>>>> directly.
> > >> >>>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>>
> > >> >>>>>>>>>>>>>
> > >> >>>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >>
> > >>
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Thanks Ash and Jarek, for the detailed comments. I largely agree with the
points mentioned by you guys, hence I updated the AIP and added a section
on Authentication, API Versioning, and Packaging as well. Please go through
it once more and let me know if there are more things to consider before I
open it for voting.
On Thu, Aug 7, 2025 at 6:03 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Also
>
> > 1. The Authentication token. How will this long lived token work without
> being insecure. Who and what will generate it? How will we identify
> top-level requests for Variables in order to be able to add Variable
> RBAC/ACLs. This is an important enough thing that I think it needs
> discussion before we vote on this AIP.
>
> We are currently discussing - in the security team - approach for JWT token
> handling, so likely we could move the discussion there, it does have some
> security implications and I think we should bring our finding to the
> devlist when we complete it, but I think we should add this case there.
> IMHO we should have a different approach for UI, different for Tasks,
> different for Triggerer, and different for DagProcessor. (possibly the
> Trigerer and DagProcessor could be the same because they share essentially
> the same long-living token. Ash - I will add this to the discussion there.
>
> J.
>
>
>
> On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Ah.. So if we are talking about a more complete approach - seeing those
> > comments from Ash - make me think if we should have another AIP.
> > (connected) about splitting the distributions. We have never finalized it
> > (nor even discussed id) but Ash - you had some initial document for that.
> > So maybe we should finalize it and rather than specify it in this AIP -
> > have a separate AIP about distribution split that AIP-92 could depend on.
> > It seems much more reasonable to split "distribution and code split" from
> > parsing isolation I think and implement them separately/in parallel.
> >
> > Reading Ash comments (and maybe I am going a bit further than Ash) it
> > calls for something that I am a big proponent of - splitting
> "airflow-core"
> > and having a different "scheduler". "webserver", "dag processor" and
> > "triggerer" distributions. Now - we have the capability of having
> "shared"
> > code - we do not need "common" code to make it happen - because we can
> > share code.
> >
> > What it could give us - on top of clean client/server split, we could
> have
> > different dependencies used by those distributions. Additionally, we
> could
> > also split-off the executors from providers and finally implement it in
> the
> > way that scheduler does not use providers at all (not even
> cncf.kubernetes
> > nor celery providers installed in scheduler nor webserver but "executors"
> > packages instead. The code sharing approach with symlinks we have now
> will
> > make it a .... breeze :) . That would also imply sharing "connection"
> > definitions through DB, and likely implementing "test connection"
> > feature properly finally (i.e executing test connection in worker /
> > triggerer rather than in web server which is a reason why we disabled it
> by
> > default now). This way "api-server" would not need any of the providers
> to
> > be installed either which IMHO is the biggest win from a security point
> of
> > view.
> >
> > And the nice thing about it is that it would be rather transparent when
> > anyone uses "pip install apache-airflow" - it would behave exactly the
> > same, no more complexity involved, simply more distributions installed
> when
> > 'apache-airflow" meta-distribution is used, but it would allow those who
> > want to implement a more complex and secure setup to have different
> > "environments" with modularized pieces of airflow installed - only
> > "apache-airflow-dag-processor + task-sdk + providers" where dag-processor
> > is run, only "apache-airflow-scheduler + executors" where scheduler is
> > installed only "apache-airflow-task-sdk + providers" where workers are
> > running, only "apache-airflow-api-server" where api-server is running and
> > only "apache-airflow-trigger + task-sdk + providers" .
> >
> > I am happy (Ash If you are fine with that) to take that original document
> > over and lead this part and new AIP to completion (including
> > implementation), I am very much convinced that this will lead to much
> > better dependency security and more modular code without impacting the
> > "apache-airflow" installation complexity.
> >
> > If we do it this way- the part of code/clean split would be "delegated
> > out" from AIP-92 to this new AIP and turned into dependency.
> >
> > J.
> >
> >
> > On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> >> This AIP is definitely heading in the right direction and is a feature
> >> I’d like to see.
> >>
> >> For me the outstanding things that need more detail:
> >>
> >> 1. The Authentication token. How will this long lived token work without
> >> being insecure. Who and what will generate it? How will we identify
> >> top-level requests for Variables in order to be able to add Variable
> >> RBAC/ACLs. This is an important enough thing that I think it needs
> >> discussion before we vote on this AIP.
> >> 2. Security generally — how will this work, especially with the
> >> multi-team? I think this likely means making the APIs work on the bundle
> >> level as you mention in the doc, but I haven’t thought deeply about this
> >> yet.
> >> 3. API Versioning? One of the the key driving goals with AIP-72 and the
> >> Task Execution SDK was the idea that “you can upgrade the API server as
> you
> >> like, and your clients/workers never need to work” — i.e. the API
> server is
> >> 100% working with all older versions of the TaskSDK. I don’t know if we
> >> will achieve that goal in the long run but it is the desire, and part of
> >> why we are using CalVer and the Cadwyn library to provide API
> versioning.
> >> 4. As mentioned previously, not sure the existing serialised JSON format
> >> for DAGs is correct, but since that now has version and we already have
> the
> >> ability to upgrade that somewhere in the Airflow Core that doesn’t
> >> necessarily become a blocker/pre-requisite for this AIP.
> >>
> >> I think Dag parsing API client+submission+parsing process manager should
> >> either live in the Task SDK dist, or in a new separate dist that uses
> >> TaskSDK, but crucially not in apache-airflow-core. My reason for this is
> >> that I want it to be possible for the server components (scheduler, API
> >> server) to not need task-sdk installed (just for cleanliness/avoiding
> >> confusion about what versions it needs) and also vice-verse, to be able
> to
> >> run a “team worker bundle” (Dag parsing, workers, triggered/async
> workers)
> >> on whatever version of TaskSDK they choose, again without
> >> apache-airflow-core installed for avoidance of doubt.
> >>
> >> Generally I would like this as it means we can have a nicer separation
> of
> >> Core and Dag parsing code, as the dag parsing itself uses the SDK, it
> would
> >> be nice to have a proper server/client split, both from a tighter
> security
> >> point-of-view, but also from a code layout point of view.
> >>
> >> -ash
> >>
> >>
> >> > On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
> >> >
> >> > Well, you started it - so it's up to you to decide if you think we
> have
> >> > consensus, or whether we need a vote.
> >> >
> >> > And It's not a question of "informal" vote but it's rather clear
> >> following
> >> > the https://www.apache.org/foundation/voting.html that we either
> need a
> >> > LAZY CONSENSUS or VOTE thread. Both are formal.
> >> >
> >> > This is the difficult part when you have a proposal, to assess (by
> you)
> >> > whether we are converging to consensus or whether vote is needed.
> There
> >> is
> >> > no other body or "authority" to do it for you.
> >> >
> >> > J.
> >> >
> >> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <
> sumeet.manit@gmail.com
> >> >
> >> > wrote:
> >> >
> >> >> Sorry for nudging again, but can we get into some consensus on this?
> I
> >> mean
> >> >> if this AIP isn't good enough, then we can drop it altogether and
> >> someone
> >> >> can rethink the whole thing. Should we do some kind of informal
> voting
> >> and
> >> >> close this thread?
> >> >>
> >> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> >>
> >> >>>>> My main concern with this right now is the serialisation format of
> >> >> DAGs
> >> >>> —
> >> >>>> it wasn’t really designed with remote submission in mind, so it
> need
> >> >> some
> >> >>>> careful examination to see if it is fit for this purpose or not.
> >> >>>
> >> >>> I understand Ash's concerns - the format has not been designed with
> >> >>> size/speed optimization in mind so **possibly** we could design a
> >> >> different
> >> >>> format that would be better suited.
> >> >>>
> >> >>> BUT ... Done is better than perfect.
> >> >>>
> >> >>> I think there are a number of risks involved in changing the format
> >> and
> >> >> it
> >> >>> could significantly increase time of development with uncertain
> gains
> >> at
> >> >>> the end - also because of the progress in compression that happened
> >> over
> >> >>> the last few years.
> >> >>>
> >> >>> It might be a good idea to experiment a bit with different
> compression
> >> >>> algorithms for "our" dag representation and possibly we could find
> the
> >> >> best
> >> >>> algorithm for "airflow dag" type of json data. There are a lot of
> >> >>> repetitions in the JSON representation and I guess in "our" json
> >> >>> representation there are some artifacts and repeated sections that
> >> simply
> >> >>> might compress well with different algorithms. Also in this case
> >> >>> speed matters (and CPU trade-off).
> >> >>>
> >> >>> Looking at compression "theory" - before we experiment with it -
> >> there is
> >> >>> the relatively new standard "zstandard"
> >> https://github.com/facebook/zstd
> >> >>> compression opensourced in 2016 which I've heard good things about -
> >> >>> especially that it maintains a very good compression rate for text
> >> data,
> >> >>> but also it is tremendously fast - especially for decompression
> >> (which is
> >> >>> super important factor for us - we compress new DAG representation
> far
> >> >> less
> >> >>> often than decompress it in general case). It is standardized in RFC
> >> >>> https://datatracker.ietf.org/doc/html/rfc8878 and there are various
> >> >>> implementations and it is even being added to Python standard
> library
> >> in
> >> >>> Python 3.14
> >> https://docs.python.org/3.14/library/compression.zstd.html
> >> >> and
> >> >>> there is a very well maintained python binding library
> >> >>> https://pypi.org/project/zstd/ to Yann Collet (algorithm author)
> >> ZSTD C
> >> >>> library. And libzstd is already part of our images - it is needed by
> >> >> other
> >> >>> dependencies of ours. All with BSD licence, directly usable by us.
> >> >>>
> >> >>> I think this one might be a good candidate for us to try, and
> possibly
> >> >> with
> >> >>> zstd we could achieve both size and CPU overhead that would be
> >> comparable
> >> >>> with any "new" format we could come up with - especially that we are
> >> >>> talking merely about processing a huge blob between "storable"
> >> >> (compressed)
> >> >>> and "locally usable" state (Python dict). We could likely use a
> >> streaming
> >> >>> JSON library (say the one that is used in Pydantic internally
> >> >>> https://github.com/pydantic/jiter - we already have it as part of
> >> >>> Pydantic)
> >> >>> to also save memory - we could stream decompressed stream into
> jitter
> >> so
> >> >>> that both the json dict and string representation does not have to
> be
> >> >>> loaded fully in memory at the same time. There are likely lots of
> >> >>> optimisations we could do - I mentioned possibly streaming the data
> >> from
> >> >>> API directly to DB (if this is possible - not sure)
> >> >>>
> >> >>> J.
> >> >>>
> >> >>>
> >> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> >> sumeet.manit@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>>>
> >> >>>>> My main concern with this right now is the serialisation format of
> >> >>> DAGs —
> >> >>>>> it wasn’t really designed with remote submission in mind, so it
> need
> >> >>> some
> >> >>>>> careful examination to see if it is fit for this purpose or not.
> >> >>>>>
> >> >>>>
> >> >>>> I'm not sure on this point, cause if we are able to convert a DAG
> >> into
> >> >>>> JSON, then it has to be transferable over the internet.
> >> >>>>
> >> >>>> In particular One of the things I worry about is that the JSON can
> >> get
> >> >>> huge
> >> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> >> >>>>
> >> >>>>
> >> >>>> Yeah, agree on this, thats why we can transfer compressed data
> >> instead
> >> >> of
> >> >>>> real json. Of course, this won't guarantee that the payload will
> >> always
> >> >>> be
> >> >>>> small enough, but we can't say that it'll definitely happen either.
> >> >>>>
> >> >>>> I also wonder if as part of this proposal we should move the
> Callback
> >> >>>>> requests off the dag parsers and on to the workers instead
> >> >>>>
> >> >>>> let's make such a "workfload" implementation stream that could
> >> support
> >> >>> both
> >> >>>>> - Deadlines and DAG parsing logic
> >> >>>>
> >> >>>>
> >> >>>> I don't have any strong opinion here, but it feels like it's gonna
> >> blow
> >> >>> up
> >> >>>> the scope of the AIP too much.
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com>
> >> wrote:
> >> >>>>
> >> >>>>>> My main concern with this right now is the serialisation format
> of
> >> >>>> DAGs —
> >> >>>>> it wasn’t really designed with remote submission in mind, so it
> need
> >> >>> some
> >> >>>>> careful examination to see if it is fit for this purpose or not.
> >> >>>>>
> >> >>>>> Yep. That might be potentially a problem (or at least "need more
> >> >>>> resources
> >> >>>>> to run airflow") and that is where my "2x memory" came from if we
> do
> >> >> it
> >> >>>> in
> >> >>>>> a trivial way. Currently we a) keep the whole DAG in memory when
> >> >>>>> serializing it b) submit it to database (also using essentially
> some
> >> >>> kind
> >> >>>>> of API (implemented by the database client) - so we know the whole
> >> >>> thing
> >> >>>>> "might work" but indeed if you use a trivial implementation of
> >> >>> submitting
> >> >>>>> the whole json - it basically means that the whole json will have
> to
> >> >>> also
> >> >>>>> be kept in the memory of API server. But we also compress it when
> >> >>> needed
> >> >>>> -
> >> >>>>> I wonder what are the compression ratios we saw with those
> 10-20MBs
> >> >>> Dags
> >> >>>> -
> >> >>>>> if the problem is using strings where bool would suffice,
> >> compression
> >> >>>>> should generally help a lot. We could only ever send compressed
> data
> >> >>> over
> >> >>>>> the API - there seems to be no need to send "plain JSON" data over
> >> >> the
> >> >>>> API
> >> >>>>> or storing the plain JSON in the DB (of course that trades memory
> >> for
> >> >>>> CPU).
> >> >>>>>
> >> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
> >> >> support
> >> >>>> for
> >> >>>>> any kind if binary data streaming - because that could help a lot
> of
> >> >> if
> >> >>>> we
> >> >>>>> could use streaming HTTP API and chunk and append the binary
> chunks
> >> >>> (when
> >> >>>>> writing) - or read data in chunks ans stream them back via the
> API.
> >> >>> That
> >> >>>>> could seriously decrease the amount of memory needed by the API
> >> >> server
> >> >>> to
> >> >>>>> process such huge serialized dags.
> >> >>>>>
> >> >>>>> And yeah - I would also love the "execute task" to be implemented
> >> >> here
> >> >>> -
> >> >>>>> but I am not sure if this should be part of the same effort or
> maybe
> >> >> a
> >> >>>>> separate implementation? That sounds very loosely coupled with DB
> >> >>>>> isolation. And it seems a common theme - I think that would also
> >> make
> >> >>> the
> >> >>>>> sync Deadline alerts case that we discussed at the dev call
> today. I
> >> >>>> wonder
> >> >>>>> if that should not be kind of parallel (let's make such a
> >> "workfload"
> >> >>>>> implementation stream that could support both - Deadlines and DAG
> >> >>> parsing
> >> >>>>> logic. We have already two "users" for it and I really love the
> >> >> saying
> >> >>>> "if
> >> >>>>> you want to make something reusable - make it usable first" -
> seems
> >> >>> like
> >> >>>>> we might have good opportunity to make such workload
> implementation
> >> >>>> "doubly
> >> >>>>> used" from the beginning which would increase chances it will be
> >> >>>>> "reusable" for other things as well :).
> >> >>>>>
> >> >>>>> J.
> >> >>>>>
> >> >>>>>
> >> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <
> ash@apache.org>
> >> >>>> wrote:
> >> >>>>>
> >> >>>>>> My main concern with this right now is the serialisation format
> of
> >> >>>> DAGs —
> >> >>>>>> it wasn’t really designed with remote submission in mind, so it
> >> >> need
> >> >>>> some
> >> >>>>>> careful examination to see if it is fit for this purpose or not.
> >> >>>>>>
> >> >>>>>> In particular One of the things I worry about is that the JSON
> can
> >> >>> get
> >> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!)
> (which
> >> >> is
> >> >>>>>> likely due to things being included as text when a bool might
> >> >>> suffice,
> >> >>>>> for
> >> >>>>>> example) But I don’t think “just submit the existing JSON over an
> >> >>> API”
> >> >>>>> is a
> >> >>>>>> good idea.
> >> >>>>>>
> >> >>>>>> I also wonder if as part of this proposal we should move the
> >> >> Callback
> >> >>>>>> requests off the dag parsers and on to the workers instead — in
> >> >>> AIP-72
> >> >>>> we
> >> >>>>>> introduced the concept of a Workload, with the only one existing
> >> >>> right
> >> >>>>> now
> >> >>>>>> is “ExecuteTask”
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> >> >>>>>> — it might be time to finally move task and dag callbacks to the
> >> >> same
> >> >>>>> thing
> >> >>>>>> and make dag parsers only responsible for, well, parsing. :)
> >> >>>>>>
> >> >>>>>> These are all solvable problems, and this will be a great feature
> >> >> to
> >> >>>>> have,
> >> >>>>>> but we need to do some more thinking and planning first.
> >> >>>>>>
> >> >>>>>> -ash
> >> >>>>>>
> >> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> >> >> sumeet.manit@gmail.com
> >> >>>>
> >> >>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>> Gentle reminder for everyone to review the proposal.
> >> >>>>>>>
> >> >>>>>>> Updated link:
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> >> >>>>>>>
> >> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> >> >>>>> sumeet.manit@gmail.com
> >> >>>>>>>
> >> >>>>>>> wrote:
> >> >>>>>>>
> >> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and others
> >> >>>>> suggested, I
> >> >>>>>>>> expanded the scope of this AIP and divided it into three
> phases.
> >> >>>> With
> >> >>>>>> the
> >> >>>>>>>> increased scope, the boundary line between this AIP and AIP-85
> >> >>> got a
> >> >>>>>> little
> >> >>>>>>>> thinner, but I believe these are still two different
> >> >> enhancements
> >> >>> to
> >> >>>>>> make.
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> >> >>>>>> sumeet.manit@gmail.com>
> >> >>>>>>>> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well to be
> >> >>> part
> >> >>>>> of
> >> >>>>>>>>> this AIP and phase out the implementation. Though I didn't
> >> >>> exclude
> >> >>>>>> Triggers
> >> >>>>>>>>> because "Uber" doesn't need that, I just thought of keeping
> the
> >> >>>> scope
> >> >>>>>> of
> >> >>>>>>>>> development small and achieving them, just like it was done in
> >> >>>>> Airlfow
> >> >>>>>> 3 by
> >> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
> >> >>>>>>>>>
> >> >>>>>>>>> But if you think Triggers should be part of this AIP itself,
> >> >>> then I
> >> >>>>> can
> >> >>>>>>>>> do that and include Triggers as well in it.
> >> >>>>>>>>>
> >> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <
> jarek@potiuk.com
> >> >>>
> >> >>>>> wrote:
> >> >>>>>>>>>
> >> >>>>>>>>>> I would very much prefer the architectural choices of this
> AIP
> >> >>> are
> >> >>>>>> based
> >> >>>>>>>>>> on
> >> >>>>>>>>>> "general public" needs rather than "Uber needs" even if Uber
> >> >>> will
> >> >>>> be
> >> >>>>>>>>>> implementing it - so from my point of view having Trigger
> >> >>>> separation
> >> >>>>>> as
> >> >>>>>>>>>> part of it is quite important.
> >> >>>>>>>>>>
> >> >>>>>>>>>> But that's not even this.
> >> >>>>>>>>>>
> >> >>>>>>>>>> We've been discussing for example for Deadlines (being
> >> >>> implemented
> >> >>>>> by
> >> >>>>>>>>>> Dennis and Ramit a possibility of short, notification-style
> >> >>>>>> "deadlines"
> >> >>>>>>>>>> to be send to triggerer for execution - this is well advanced
> >> >>> now,
> >> >>>>> and
> >> >>>>>>>>>> whether you want it or not Dag-provided code might be
> >> >> serialized
> >> >>>> and
> >> >>>>>> sent
> >> >>>>>>>>>> to triggerer for execution. This is part of our "broader"
> >> >>>>>> architectural
> >> >>>>>>>>>> change where we treat "workers" and "triggerer" similarly as
> a
> >> >>>>> general
> >> >>>>>>>>>> executors of "sync" and "async" tasks respectively. That's
> >> >> where
> >> >>>>>> Airflow
> >> >>>>>>>>>> is
> >> >>>>>>>>>> evolving towards - inevitably.
> >> >>>>>>>>>>
> >> >>>>>>>>>> But we can of course phase things in out for implementation -
> >> >>> even
> >> >>>>> if
> >> >>>>>> AIP
> >> >>>>>>>>>> should cover both, I think if the goal of the AIP and
> preamble
> >> >>> is
> >> >>>>>> about
> >> >>>>>>>>>> separating "user code" from "database" as the main reason, it
> >> >>> also
> >> >>>>>> means
> >> >>>>>>>>>> Triggerer if you ask me (from design point of view at least).
> >> >>>>>>>>>>
> >> >>>>>>>>>> Again implementation can be phased and even different people
> >> >> and
> >> >>>>> teams
> >> >>>>>>>>>> might work on those phases/pieces.
> >> >>>>>>>>>>
> >> >>>>>>>>>> J.
> >> >>>>>>>>>>
> >> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> >> >>>>>> sumeet.manit@gmail.com
> >> >>>>>>>>>>>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers
> as
> >> >>>> well,
> >> >>>>>>>>>> but
> >> >>>>>>>>>>>> that
> >> >>>>>>>>>>>> can be done as part of a different AIP
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> You won't achieve your goal of "true" isolation of user code
> >> >> if
> >> >>>> you
> >> >>>>>>>>>> don't
> >> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it - it
> >> >> should
> >> >>>>> cover
> >> >>>>>>>>>>> both.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> My bad, should've explained our architecture for triggers as
> >> >>>> well,
> >> >>>>>>>>>>> apologies. So here it is:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> - Triggers would be running on a centralized service, so
> >> >> all
> >> >>>> the
> >> >>>>>>>>>> Trigger
> >> >>>>>>>>>>> classes will be part of the platform team's repo and not
> >> >> the
> >> >>>>>>>>>> customer's
> >> >>>>>>>>>>> repo
> >> >>>>>>>>>>> - The triggers won't be able to use any libs other than std
> >> >>>> ones,
> >> >>>>>>>>>> which
> >> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> >> >>>>>>>>>>> - As we are the owners of the core Airflow repo, customers
> >> >>> have
> >> >>>>> to
> >> >>>>>>>>>> get
> >> >>>>>>>>>>> our approval to land any class in this path (unlike the
> >> >> dags
> >> >>>> repo
> >> >>>>>>>>>> which
> >> >>>>>>>>>>> they own)
> >> >>>>>>>>>>> - When a customer's task defer, we would have an allowlist
> >> >> on
> >> >>>> our
> >> >>>>>>>>>> side
> >> >>>>>>>>>>> to check if we should do the async polling or not
> >> >>>>>>>>>>> - If the Trigger class isn't part of our repo (allowlist),
> >> >>> just
> >> >>>>>>>>>> fail the
> >> >>>>>>>>>>> task, as anyway we won't be having the code that they used
> >> >> in
> >> >>>> the
> >> >>>>>>>>>>> trigger
> >> >>>>>>>>>>> class
> >> >>>>>>>>>>> - If any of these conditions aren't suitable for you (as a
> >> >>>>>>>>>> customer),
> >> >>>>>>>>>>> feel free to use sync tasks only
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> >> >> communicate
> >> >>>> over
> >> >>>>>>>>>> apis
> >> >>>>>>>>>>> only. If that is done, then we can have instances of
> >> >> triggerer
> >> >>>> svc
> >> >>>>>>>>>> running
> >> >>>>>>>>>>> at customer's side as well, which can process any type of
> >> >>> trigger
> >> >>>>>>>>>> class.
> >> >>>>>>>>>>> Though that's not a blocker for us at the moment, cause
> >> >>> triggerer
> >> >>>>> are
> >> >>>>>>>>>>> mostly doing just polling using simple libs like requests.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> >> >>>>>>>>>> <ikholopov@google.com.invalid
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I believe
> it
> >> >>>>> aligns
> >> >>>>>>>>>> well
> >> >>>>>>>>>>>> with the goals of making Airflow well-scalable beyond a
> >> >>>>> single-team
> >> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full support
> >> >>> with
> >> >>>>> this
> >> >>>>>>>>>> one.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> I've left a couple of clarification requests on the AIP
> >> >> page.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Thanks,
> >> >>>>>>>>>>>> Igor
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> >> >>>>>>>>>>> sumeet.manit@gmail.com>
> >> >>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's good to
> >> >>> know
> >> >>>>>>>>>> that
> >> >>>>>>>>>>> the
> >> >>>>>>>>>>>>> DAG processor has some preemptive measures in place to
> >> >>> prevent
> >> >>>>>>>>>> access
> >> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to solve
> >> >> is
> >> >>>> not
> >> >>>>> to
> >> >>>>>>>>>>>> provide
> >> >>>>>>>>>>>>> DB creds to the customer teams, who are using Airflow as a
> >> >>>>>>>>>> multi-tenant
> >> >>>>>>>>>>>>> orchestration platform. I've updated the doc to reflect
> >> >> this
> >> >>>>> point
> >> >>>>>>>>>> as
> >> >>>>>>>>>>>> well.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Answering Jarek's points,
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism, added
> >> >>> that
> >> >>>>> in
> >> >>>>>>>>>> doc,
> >> >>>>>>>>>>>> but
> >> >>>>>>>>>>>>> still how the token can be obtained (safely) is still open
> >> >> in
> >> >>>> my
> >> >>>>>>>>>> mind.
> >> >>>>>>>>>>> I
> >> >>>>>>>>>>>>> believe the token used by task executors can be created
> >> >>> outside
> >> >>>>> of
> >> >>>>>>>>>> it
> >> >>>>>>>>>>> as
> >> >>>>>>>>>>>>> well (I may be wrong here).
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers
> as
> >> >>>> well,
> >> >>>>>>>>>> but
> >> >>>>>>>>>>>> that
> >> >>>>>>>>>>>>> can be done as part of a different AIP
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work largely.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can
> >> >>> work
> >> >>>>>>>>>> with
> >> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would be
> >> >> treated
> >> >>>> as
> >> >>>>> a
> >> >>>>>>>>>> diff
> >> >>>>>>>>>>>>> bundle.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Also, added points around callbacks, as these are also
> >> >>> fetched
> >> >>>>>>>>>> directly
> >> >>>>>>>>>>>>> from the DB.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> >> >>>> jarek@potiuk.com>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
> >> >>> not
> >> >>>>>>>>>>>> protection
> >> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> >> >>> protect
> >> >>>>>>>>>> against
> >> >>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
> >> >> in
> >> >>>> the
> >> >>>>>>>>>>>> parsing
> >> >>>>>>>>>>>>>> process , as are the env var configs
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >> >>>>>>>>>>>>>> -
> >> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no protection
> >> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed back
> >> >> then
> >> >>> in
> >> >>>>>>>>>> March
> >> >>>>>>>>>>> I
> >> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with full
> >> >>>> knowledge
> >> >>>>>>>>>> it's
> >> >>>>>>>>>>>> not
> >> >>>>>>>>>>>>>> protecting against all threats.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> >> >>>>>>>>>> ash@apache.org>
> >> >>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
> >> >>> not
> >> >>>>>>>>>>>> protection
> >> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> >> >>> protect
> >> >>>>>>>>>>> against
> >> >>>>>>>>>>>>>>> casual DB access attempts - the db session is blanked
> out
> >> >>> in
> >> >>>>>>>>>> the
> >> >>>>>>>>>>>>> parsing
> >> >>>>>>>>>>>>>>> process , as are the env var configs
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> >> >> protection
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> >> >> jarek@potiuk.com>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no
> built-in
> >> >>>>>>>>>>> protection
> >> >>>>>>>>>>>>>>> against
> >> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example - read
> >> >>> database
> >> >>>>>>>>>>>>>>>> credentials from airflow configuration and use them to
> >> >>> talk
> >> >>>>>>>>>> to DB
> >> >>>>>>>>>>>>>>> directly.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >>
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
Also
> 1. The Authentication token. How will this long lived token work without
being insecure. Who and what will generate it? How will we identify
top-level requests for Variables in order to be able to add Variable
RBAC/ACLs. This is an important enough thing that I think it needs
discussion before we vote on this AIP.
We are currently discussing - in the security team - approach for JWT token
handling, so likely we could move the discussion there, it does have some
security implications and I think we should bring our finding to the
devlist when we complete it, but I think we should add this case there.
IMHO we should have a different approach for UI, different for Tasks,
different for Triggerer, and different for DagProcessor. (possibly the
Trigerer and DagProcessor could be the same because they share essentially
the same long-living token. Ash - I will add this to the discussion there.
J.
On Thu, Aug 7, 2025 at 2:23 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Ah.. So if we are talking about a more complete approach - seeing those
> comments from Ash - make me think if we should have another AIP.
> (connected) about splitting the distributions. We have never finalized it
> (nor even discussed id) but Ash - you had some initial document for that.
> So maybe we should finalize it and rather than specify it in this AIP -
> have a separate AIP about distribution split that AIP-92 could depend on.
> It seems much more reasonable to split "distribution and code split" from
> parsing isolation I think and implement them separately/in parallel.
>
> Reading Ash comments (and maybe I am going a bit further than Ash) it
> calls for something that I am a big proponent of - splitting "airflow-core"
> and having a different "scheduler". "webserver", "dag processor" and
> "triggerer" distributions. Now - we have the capability of having "shared"
> code - we do not need "common" code to make it happen - because we can
> share code.
>
> What it could give us - on top of clean client/server split, we could have
> different dependencies used by those distributions. Additionally, we could
> also split-off the executors from providers and finally implement it in the
> way that scheduler does not use providers at all (not even cncf.kubernetes
> nor celery providers installed in scheduler nor webserver but "executors"
> packages instead. The code sharing approach with symlinks we have now will
> make it a .... breeze :) . That would also imply sharing "connection"
> definitions through DB, and likely implementing "test connection"
> feature properly finally (i.e executing test connection in worker /
> triggerer rather than in web server which is a reason why we disabled it by
> default now). This way "api-server" would not need any of the providers to
> be installed either which IMHO is the biggest win from a security point of
> view.
>
> And the nice thing about it is that it would be rather transparent when
> anyone uses "pip install apache-airflow" - it would behave exactly the
> same, no more complexity involved, simply more distributions installed when
> 'apache-airflow" meta-distribution is used, but it would allow those who
> want to implement a more complex and secure setup to have different
> "environments" with modularized pieces of airflow installed - only
> "apache-airflow-dag-processor + task-sdk + providers" where dag-processor
> is run, only "apache-airflow-scheduler + executors" where scheduler is
> installed only "apache-airflow-task-sdk + providers" where workers are
> running, only "apache-airflow-api-server" where api-server is running and
> only "apache-airflow-trigger + task-sdk + providers" .
>
> I am happy (Ash If you are fine with that) to take that original document
> over and lead this part and new AIP to completion (including
> implementation), I am very much convinced that this will lead to much
> better dependency security and more modular code without impacting the
> "apache-airflow" installation complexity.
>
> If we do it this way- the part of code/clean split would be "delegated
> out" from AIP-92 to this new AIP and turned into dependency.
>
> J.
>
>
> On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> This AIP is definitely heading in the right direction and is a feature
>> I’d like to see.
>>
>> For me the outstanding things that need more detail:
>>
>> 1. The Authentication token. How will this long lived token work without
>> being insecure. Who and what will generate it? How will we identify
>> top-level requests for Variables in order to be able to add Variable
>> RBAC/ACLs. This is an important enough thing that I think it needs
>> discussion before we vote on this AIP.
>> 2. Security generally — how will this work, especially with the
>> multi-team? I think this likely means making the APIs work on the bundle
>> level as you mention in the doc, but I haven’t thought deeply about this
>> yet.
>> 3. API Versioning? One of the the key driving goals with AIP-72 and the
>> Task Execution SDK was the idea that “you can upgrade the API server as you
>> like, and your clients/workers never need to work” — i.e. the API server is
>> 100% working with all older versions of the TaskSDK. I don’t know if we
>> will achieve that goal in the long run but it is the desire, and part of
>> why we are using CalVer and the Cadwyn library to provide API versioning.
>> 4. As mentioned previously, not sure the existing serialised JSON format
>> for DAGs is correct, but since that now has version and we already have the
>> ability to upgrade that somewhere in the Airflow Core that doesn’t
>> necessarily become a blocker/pre-requisite for this AIP.
>>
>> I think Dag parsing API client+submission+parsing process manager should
>> either live in the Task SDK dist, or in a new separate dist that uses
>> TaskSDK, but crucially not in apache-airflow-core. My reason for this is
>> that I want it to be possible for the server components (scheduler, API
>> server) to not need task-sdk installed (just for cleanliness/avoiding
>> confusion about what versions it needs) and also vice-verse, to be able to
>> run a “team worker bundle” (Dag parsing, workers, triggered/async workers)
>> on whatever version of TaskSDK they choose, again without
>> apache-airflow-core installed for avoidance of doubt.
>>
>> Generally I would like this as it means we can have a nicer separation of
>> Core and Dag parsing code, as the dag parsing itself uses the SDK, it would
>> be nice to have a proper server/client split, both from a tighter security
>> point-of-view, but also from a code layout point of view.
>>
>> -ash
>>
>>
>> > On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > Well, you started it - so it's up to you to decide if you think we have
>> > consensus, or whether we need a vote.
>> >
>> > And It's not a question of "informal" vote but it's rather clear
>> following
>> > the https://www.apache.org/foundation/voting.html that we either need a
>> > LAZY CONSENSUS or VOTE thread. Both are formal.
>> >
>> > This is the difficult part when you have a proposal, to assess (by you)
>> > whether we are converging to consensus or whether vote is needed. There
>> is
>> > no other body or "authority" to do it for you.
>> >
>> > J.
>> >
>> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <sumeet.manit@gmail.com
>> >
>> > wrote:
>> >
>> >> Sorry for nudging again, but can we get into some consensus on this? I
>> mean
>> >> if this AIP isn't good enough, then we can drop it altogether and
>> someone
>> >> can rethink the whole thing. Should we do some kind of informal voting
>> and
>> >> close this thread?
>> >>
>> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>
>> >>>>> My main concern with this right now is the serialisation format of
>> >> DAGs
>> >>> —
>> >>>> it wasn’t really designed with remote submission in mind, so it need
>> >> some
>> >>>> careful examination to see if it is fit for this purpose or not.
>> >>>
>> >>> I understand Ash's concerns - the format has not been designed with
>> >>> size/speed optimization in mind so **possibly** we could design a
>> >> different
>> >>> format that would be better suited.
>> >>>
>> >>> BUT ... Done is better than perfect.
>> >>>
>> >>> I think there are a number of risks involved in changing the format
>> and
>> >> it
>> >>> could significantly increase time of development with uncertain gains
>> at
>> >>> the end - also because of the progress in compression that happened
>> over
>> >>> the last few years.
>> >>>
>> >>> It might be a good idea to experiment a bit with different compression
>> >>> algorithms for "our" dag representation and possibly we could find the
>> >> best
>> >>> algorithm for "airflow dag" type of json data. There are a lot of
>> >>> repetitions in the JSON representation and I guess in "our" json
>> >>> representation there are some artifacts and repeated sections that
>> simply
>> >>> might compress well with different algorithms. Also in this case
>> >>> speed matters (and CPU trade-off).
>> >>>
>> >>> Looking at compression "theory" - before we experiment with it -
>> there is
>> >>> the relatively new standard "zstandard"
>> https://github.com/facebook/zstd
>> >>> compression opensourced in 2016 which I've heard good things about -
>> >>> especially that it maintains a very good compression rate for text
>> data,
>> >>> but also it is tremendously fast - especially for decompression
>> (which is
>> >>> super important factor for us - we compress new DAG representation far
>> >> less
>> >>> often than decompress it in general case). It is standardized in RFC
>> >>> https://datatracker.ietf.org/doc/html/rfc8878 and there are various
>> >>> implementations and it is even being added to Python standard library
>> in
>> >>> Python 3.14
>> https://docs.python.org/3.14/library/compression.zstd.html
>> >> and
>> >>> there is a very well maintained python binding library
>> >>> https://pypi.org/project/zstd/ to Yann Collet (algorithm author)
>> ZSTD C
>> >>> library. And libzstd is already part of our images - it is needed by
>> >> other
>> >>> dependencies of ours. All with BSD licence, directly usable by us.
>> >>>
>> >>> I think this one might be a good candidate for us to try, and possibly
>> >> with
>> >>> zstd we could achieve both size and CPU overhead that would be
>> comparable
>> >>> with any "new" format we could come up with - especially that we are
>> >>> talking merely about processing a huge blob between "storable"
>> >> (compressed)
>> >>> and "locally usable" state (Python dict). We could likely use a
>> streaming
>> >>> JSON library (say the one that is used in Pydantic internally
>> >>> https://github.com/pydantic/jiter - we already have it as part of
>> >>> Pydantic)
>> >>> to also save memory - we could stream decompressed stream into jitter
>> so
>> >>> that both the json dict and string representation does not have to be
>> >>> loaded fully in memory at the same time. There are likely lots of
>> >>> optimisations we could do - I mentioned possibly streaming the data
>> from
>> >>> API directly to DB (if this is possible - not sure)
>> >>>
>> >>> J.
>> >>>
>> >>>
>> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
>> sumeet.manit@gmail.com>
>> >>> wrote:
>> >>>
>> >>>>>
>> >>>>> My main concern with this right now is the serialisation format of
>> >>> DAGs —
>> >>>>> it wasn’t really designed with remote submission in mind, so it need
>> >>> some
>> >>>>> careful examination to see if it is fit for this purpose or not.
>> >>>>>
>> >>>>
>> >>>> I'm not sure on this point, cause if we are able to convert a DAG
>> into
>> >>>> JSON, then it has to be transferable over the internet.
>> >>>>
>> >>>> In particular One of the things I worry about is that the JSON can
>> get
>> >>> huge
>> >>>>> — I’ve seem this as large as 10-20Mb for some dags
>> >>>>
>> >>>>
>> >>>> Yeah, agree on this, thats why we can transfer compressed data
>> instead
>> >> of
>> >>>> real json. Of course, this won't guarantee that the payload will
>> always
>> >>> be
>> >>>> small enough, but we can't say that it'll definitely happen either.
>> >>>>
>> >>>> I also wonder if as part of this proposal we should move the Callback
>> >>>>> requests off the dag parsers and on to the workers instead
>> >>>>
>> >>>> let's make such a "workfload" implementation stream that could
>> support
>> >>> both
>> >>>>> - Deadlines and DAG parsing logic
>> >>>>
>> >>>>
>> >>>> I don't have any strong opinion here, but it feels like it's gonna
>> blow
>> >>> up
>> >>>> the scope of the AIP too much.
>> >>>>
>> >>>>
>> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> >>>>
>> >>>>>> My main concern with this right now is the serialisation format of
>> >>>> DAGs —
>> >>>>> it wasn’t really designed with remote submission in mind, so it need
>> >>> some
>> >>>>> careful examination to see if it is fit for this purpose or not.
>> >>>>>
>> >>>>> Yep. That might be potentially a problem (or at least "need more
>> >>>> resources
>> >>>>> to run airflow") and that is where my "2x memory" came from if we do
>> >> it
>> >>>> in
>> >>>>> a trivial way. Currently we a) keep the whole DAG in memory when
>> >>>>> serializing it b) submit it to database (also using essentially some
>> >>> kind
>> >>>>> of API (implemented by the database client) - so we know the whole
>> >>> thing
>> >>>>> "might work" but indeed if you use a trivial implementation of
>> >>> submitting
>> >>>>> the whole json - it basically means that the whole json will have to
>> >>> also
>> >>>>> be kept in the memory of API server. But we also compress it when
>> >>> needed
>> >>>> -
>> >>>>> I wonder what are the compression ratios we saw with those 10-20MBs
>> >>> Dags
>> >>>> -
>> >>>>> if the problem is using strings where bool would suffice,
>> compression
>> >>>>> should generally help a lot. We could only ever send compressed data
>> >>> over
>> >>>>> the API - there seems to be no need to send "plain JSON" data over
>> >> the
>> >>>> API
>> >>>>> or storing the plain JSON in the DB (of course that trades memory
>> for
>> >>>> CPU).
>> >>>>>
>> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
>> >> support
>> >>>> for
>> >>>>> any kind if binary data streaming - because that could help a lot of
>> >> if
>> >>>> we
>> >>>>> could use streaming HTTP API and chunk and append the binary chunks
>> >>> (when
>> >>>>> writing) - or read data in chunks ans stream them back via the API.
>> >>> That
>> >>>>> could seriously decrease the amount of memory needed by the API
>> >> server
>> >>> to
>> >>>>> process such huge serialized dags.
>> >>>>>
>> >>>>> And yeah - I would also love the "execute task" to be implemented
>> >> here
>> >>> -
>> >>>>> but I am not sure if this should be part of the same effort or maybe
>> >> a
>> >>>>> separate implementation? That sounds very loosely coupled with DB
>> >>>>> isolation. And it seems a common theme - I think that would also
>> make
>> >>> the
>> >>>>> sync Deadline alerts case that we discussed at the dev call today. I
>> >>>> wonder
>> >>>>> if that should not be kind of parallel (let's make such a
>> "workfload"
>> >>>>> implementation stream that could support both - Deadlines and DAG
>> >>> parsing
>> >>>>> logic. We have already two "users" for it and I really love the
>> >> saying
>> >>>> "if
>> >>>>> you want to make something reusable - make it usable first" - seems
>> >>> like
>> >>>>> we might have good opportunity to make such workload implementation
>> >>>> "doubly
>> >>>>> used" from the beginning which would increase chances it will be
>> >>>>> "reusable" for other things as well :).
>> >>>>>
>> >>>>> J.
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org>
>> >>>> wrote:
>> >>>>>
>> >>>>>> My main concern with this right now is the serialisation format of
>> >>>> DAGs —
>> >>>>>> it wasn’t really designed with remote submission in mind, so it
>> >> need
>> >>>> some
>> >>>>>> careful examination to see if it is fit for this purpose or not.
>> >>>>>>
>> >>>>>> In particular One of the things I worry about is that the JSON can
>> >>> get
>> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which
>> >> is
>> >>>>>> likely due to things being included as text when a bool might
>> >>> suffice,
>> >>>>> for
>> >>>>>> example) But I don’t think “just submit the existing JSON over an
>> >>> API”
>> >>>>> is a
>> >>>>>> good idea.
>> >>>>>>
>> >>>>>> I also wonder if as part of this proposal we should move the
>> >> Callback
>> >>>>>> requests off the dag parsers and on to the workers instead — in
>> >>> AIP-72
>> >>>> we
>> >>>>>> introduced the concept of a Workload, with the only one existing
>> >>> right
>> >>>>> now
>> >>>>>> is “ExecuteTask”
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
>> >>>>>> — it might be time to finally move task and dag callbacks to the
>> >> same
>> >>>>> thing
>> >>>>>> and make dag parsers only responsible for, well, parsing. :)
>> >>>>>>
>> >>>>>> These are all solvable problems, and this will be a great feature
>> >> to
>> >>>>> have,
>> >>>>>> but we need to do some more thinking and planning first.
>> >>>>>>
>> >>>>>> -ash
>> >>>>>>
>> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
>> >> sumeet.manit@gmail.com
>> >>>>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Gentle reminder for everyone to review the proposal.
>> >>>>>>>
>> >>>>>>> Updated link:
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
>> >>>>>>>
>> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
>> >>>>> sumeet.manit@gmail.com
>> >>>>>>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and others
>> >>>>> suggested, I
>> >>>>>>>> expanded the scope of this AIP and divided it into three phases.
>> >>>> With
>> >>>>>> the
>> >>>>>>>> increased scope, the boundary line between this AIP and AIP-85
>> >>> got a
>> >>>>>> little
>> >>>>>>>> thinner, but I believe these are still two different
>> >> enhancements
>> >>> to
>> >>>>>> make.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
>> >>>>>> sumeet.manit@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well to be
>> >>> part
>> >>>>> of
>> >>>>>>>>> this AIP and phase out the implementation. Though I didn't
>> >>> exclude
>> >>>>>> Triggers
>> >>>>>>>>> because "Uber" doesn't need that, I just thought of keeping the
>> >>>> scope
>> >>>>>> of
>> >>>>>>>>> development small and achieving them, just like it was done in
>> >>>>> Airlfow
>> >>>>>> 3 by
>> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
>> >>>>>>>>>
>> >>>>>>>>> But if you think Triggers should be part of this AIP itself,
>> >>> then I
>> >>>>> can
>> >>>>>>>>> do that and include Triggers as well in it.
>> >>>>>>>>>
>> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <jarek@potiuk.com
>> >>>
>> >>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> I would very much prefer the architectural choices of this AIP
>> >>> are
>> >>>>>> based
>> >>>>>>>>>> on
>> >>>>>>>>>> "general public" needs rather than "Uber needs" even if Uber
>> >>> will
>> >>>> be
>> >>>>>>>>>> implementing it - so from my point of view having Trigger
>> >>>> separation
>> >>>>>> as
>> >>>>>>>>>> part of it is quite important.
>> >>>>>>>>>>
>> >>>>>>>>>> But that's not even this.
>> >>>>>>>>>>
>> >>>>>>>>>> We've been discussing for example for Deadlines (being
>> >>> implemented
>> >>>>> by
>> >>>>>>>>>> Dennis and Ramit a possibility of short, notification-style
>> >>>>>> "deadlines"
>> >>>>>>>>>> to be send to triggerer for execution - this is well advanced
>> >>> now,
>> >>>>> and
>> >>>>>>>>>> whether you want it or not Dag-provided code might be
>> >> serialized
>> >>>> and
>> >>>>>> sent
>> >>>>>>>>>> to triggerer for execution. This is part of our "broader"
>> >>>>>> architectural
>> >>>>>>>>>> change where we treat "workers" and "triggerer" similarly as a
>> >>>>> general
>> >>>>>>>>>> executors of "sync" and "async" tasks respectively. That's
>> >> where
>> >>>>>> Airflow
>> >>>>>>>>>> is
>> >>>>>>>>>> evolving towards - inevitably.
>> >>>>>>>>>>
>> >>>>>>>>>> But we can of course phase things in out for implementation -
>> >>> even
>> >>>>> if
>> >>>>>> AIP
>> >>>>>>>>>> should cover both, I think if the goal of the AIP and preamble
>> >>> is
>> >>>>>> about
>> >>>>>>>>>> separating "user code" from "database" as the main reason, it
>> >>> also
>> >>>>>> means
>> >>>>>>>>>> Triggerer if you ask me (from design point of view at least).
>> >>>>>>>>>>
>> >>>>>>>>>> Again implementation can be phased and even different people
>> >> and
>> >>>>> teams
>> >>>>>>>>>> might work on those phases/pieces.
>> >>>>>>>>>>
>> >>>>>>>>>> J.
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
>> >>>>>> sumeet.manit@gmail.com
>> >>>>>>>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers as
>> >>>> well,
>> >>>>>>>>>> but
>> >>>>>>>>>>>> that
>> >>>>>>>>>>>> can be done as part of a different AIP
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> You won't achieve your goal of "true" isolation of user code
>> >> if
>> >>>> you
>> >>>>>>>>>> don't
>> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it - it
>> >> should
>> >>>>> cover
>> >>>>>>>>>>> both.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> My bad, should've explained our architecture for triggers as
>> >>>> well,
>> >>>>>>>>>>> apologies. So here it is:
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> - Triggers would be running on a centralized service, so
>> >> all
>> >>>> the
>> >>>>>>>>>> Trigger
>> >>>>>>>>>>> classes will be part of the platform team's repo and not
>> >> the
>> >>>>>>>>>> customer's
>> >>>>>>>>>>> repo
>> >>>>>>>>>>> - The triggers won't be able to use any libs other than std
>> >>>> ones,
>> >>>>>>>>>> which
>> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
>> >>>>>>>>>>> - As we are the owners of the core Airflow repo, customers
>> >>> have
>> >>>>> to
>> >>>>>>>>>> get
>> >>>>>>>>>>> our approval to land any class in this path (unlike the
>> >> dags
>> >>>> repo
>> >>>>>>>>>> which
>> >>>>>>>>>>> they own)
>> >>>>>>>>>>> - When a customer's task defer, we would have an allowlist
>> >> on
>> >>>> our
>> >>>>>>>>>> side
>> >>>>>>>>>>> to check if we should do the async polling or not
>> >>>>>>>>>>> - If the Trigger class isn't part of our repo (allowlist),
>> >>> just
>> >>>>>>>>>> fail the
>> >>>>>>>>>>> task, as anyway we won't be having the code that they used
>> >> in
>> >>>> the
>> >>>>>>>>>>> trigger
>> >>>>>>>>>>> class
>> >>>>>>>>>>> - If any of these conditions aren't suitable for you (as a
>> >>>>>>>>>> customer),
>> >>>>>>>>>>> feel free to use sync tasks only
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> But in general, I agree to make triggerer svc also
>> >> communicate
>> >>>> over
>> >>>>>>>>>> apis
>> >>>>>>>>>>> only. If that is done, then we can have instances of
>> >> triggerer
>> >>>> svc
>> >>>>>>>>>> running
>> >>>>>>>>>>> at customer's side as well, which can process any type of
>> >>> trigger
>> >>>>>>>>>> class.
>> >>>>>>>>>>> Though that's not a blocker for us at the moment, cause
>> >>> triggerer
>> >>>>> are
>> >>>>>>>>>>> mostly doing just polling using simple libs like requests.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
>> >>>>>>>>>> <ikholopov@google.com.invalid
>> >>>>>>>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
>> >>>>> aligns
>> >>>>>>>>>> well
>> >>>>>>>>>>>> with the goals of making Airflow well-scalable beyond a
>> >>>>> single-team
>> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full support
>> >>> with
>> >>>>> this
>> >>>>>>>>>> one.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I've left a couple of clarification requests on the AIP
>> >> page.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Igor
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
>> >>>>>>>>>>> sumeet.manit@gmail.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's good to
>> >>> know
>> >>>>>>>>>> that
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>> DAG processor has some preemptive measures in place to
>> >>> prevent
>> >>>>>>>>>> access
>> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to solve
>> >> is
>> >>>> not
>> >>>>> to
>> >>>>>>>>>>>> provide
>> >>>>>>>>>>>>> DB creds to the customer teams, who are using Airflow as a
>> >>>>>>>>>> multi-tenant
>> >>>>>>>>>>>>> orchestration platform. I've updated the doc to reflect
>> >> this
>> >>>>> point
>> >>>>>>>>>> as
>> >>>>>>>>>>>> well.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Answering Jarek's points,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism, added
>> >>> that
>> >>>>> in
>> >>>>>>>>>> doc,
>> >>>>>>>>>>>> but
>> >>>>>>>>>>>>> still how the token can be obtained (safely) is still open
>> >> in
>> >>>> my
>> >>>>>>>>>> mind.
>> >>>>>>>>>>> I
>> >>>>>>>>>>>>> believe the token used by task executors can be created
>> >>> outside
>> >>>>> of
>> >>>>>>>>>> it
>> >>>>>>>>>>> as
>> >>>>>>>>>>>>> well (I may be wrong here).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers as
>> >>>> well,
>> >>>>>>>>>> but
>> >>>>>>>>>>>> that
>> >>>>>>>>>>>>> can be done as part of a different AIP
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work largely.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can
>> >>> work
>> >>>>>>>>>> with
>> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would be
>> >> treated
>> >>>> as
>> >>>>> a
>> >>>>>>>>>> diff
>> >>>>>>>>>>>>> bundle.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Also, added points around callbacks, as these are also
>> >>> fetched
>> >>>>>>>>>> directly
>> >>>>>>>>>>>>> from the DB.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
>> >>>> jarek@potiuk.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
>> >>> not
>> >>>>>>>>>>>> protection
>> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
>> >>> protect
>> >>>>>>>>>> against
>> >>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
>> >> in
>> >>>> the
>> >>>>>>>>>>>> parsing
>> >>>>>>>>>>>>>> process , as are the env var configs
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>> >>>>>>>>>>>>>> -
>> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no protection
>> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed back
>> >> then
>> >>> in
>> >>>>>>>>>> March
>> >>>>>>>>>>> I
>> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with full
>> >>>> knowledge
>> >>>>>>>>>> it's
>> >>>>>>>>>>>> not
>> >>>>>>>>>>>>>> protecting against all threats.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
>> >>>>>>>>>> ash@apache.org>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
>> >>> not
>> >>>>>>>>>>>> protection
>> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
>> >>> protect
>> >>>>>>>>>>> against
>> >>>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
>> >>> in
>> >>>>>>>>>> the
>> >>>>>>>>>>>>> parsing
>> >>>>>>>>>>>>>>> process , as are the env var configs
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
>> >> protection
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
>> >> jarek@potiuk.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no built-in
>> >>>>>>>>>>> protection
>> >>>>>>>>>>>>>>> against
>> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example - read
>> >>> database
>> >>>>>>>>>>>>>>>> credentials from airflow configuration and use them to
>> >>> talk
>> >>>>>>>>>> to DB
>> >>>>>>>>>>>>>>> directly.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
Ah.. So if we are talking about a more complete approach - seeing those
comments from Ash - make me think if we should have another AIP.
(connected) about splitting the distributions. We have never finalized it
(nor even discussed id) but Ash - you had some initial document for that.
So maybe we should finalize it and rather than specify it in this AIP -
have a separate AIP about distribution split that AIP-92 could depend on.
It seems much more reasonable to split "distribution and code split" from
parsing isolation I think and implement them separately/in parallel.
Reading Ash comments (and maybe I am going a bit further than Ash) it calls
for something that I am a big proponent of - splitting "airflow-core" and
having a different "scheduler". "webserver", "dag processor" and
"triggerer" distributions. Now - we have the capability of having "shared"
code - we do not need "common" code to make it happen - because we can
share code.
What it could give us - on top of clean client/server split, we could have
different dependencies used by those distributions. Additionally, we could
also split-off the executors from providers and finally implement it in the
way that scheduler does not use providers at all (not even cncf.kubernetes
nor celery providers installed in scheduler nor webserver but "executors"
packages instead. The code sharing approach with symlinks we have now will
make it a .... breeze :) . That would also imply sharing "connection"
definitions through DB, and likely implementing "test connection"
feature properly finally (i.e executing test connection in worker /
triggerer rather than in web server which is a reason why we disabled it by
default now). This way "api-server" would not need any of the providers to
be installed either which IMHO is the biggest win from a security point of
view.
And the nice thing about it is that it would be rather transparent when
anyone uses "pip install apache-airflow" - it would behave exactly the
same, no more complexity involved, simply more distributions installed when
'apache-airflow" meta-distribution is used, but it would allow those who
want to implement a more complex and secure setup to have different
"environments" with modularized pieces of airflow installed - only
"apache-airflow-dag-processor + task-sdk + providers" where dag-processor
is run, only "apache-airflow-scheduler + executors" where scheduler is
installed only "apache-airflow-task-sdk + providers" where workers are
running, only "apache-airflow-api-server" where api-server is running and
only "apache-airflow-trigger + task-sdk + providers" .
I am happy (Ash If you are fine with that) to take that original document
over and lead this part and new AIP to completion (including
implementation), I am very much convinced that this will lead to much
better dependency security and more modular code without impacting the
"apache-airflow" installation complexity.
If we do it this way- the part of code/clean split would be "delegated out"
from AIP-92 to this new AIP and turned into dependency.
J.
On Thu, Aug 7, 2025 at 1:51 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> This AIP is definitely heading in the right direction and is a feature I’d
> like to see.
>
> For me the outstanding things that need more detail:
>
> 1. The Authentication token. How will this long lived token work without
> being insecure. Who and what will generate it? How will we identify
> top-level requests for Variables in order to be able to add Variable
> RBAC/ACLs. This is an important enough thing that I think it needs
> discussion before we vote on this AIP.
> 2. Security generally — how will this work, especially with the
> multi-team? I think this likely means making the APIs work on the bundle
> level as you mention in the doc, but I haven’t thought deeply about this
> yet.
> 3. API Versioning? One of the the key driving goals with AIP-72 and the
> Task Execution SDK was the idea that “you can upgrade the API server as you
> like, and your clients/workers never need to work” — i.e. the API server is
> 100% working with all older versions of the TaskSDK. I don’t know if we
> will achieve that goal in the long run but it is the desire, and part of
> why we are using CalVer and the Cadwyn library to provide API versioning.
> 4. As mentioned previously, not sure the existing serialised JSON format
> for DAGs is correct, but since that now has version and we already have the
> ability to upgrade that somewhere in the Airflow Core that doesn’t
> necessarily become a blocker/pre-requisite for this AIP.
>
> I think Dag parsing API client+submission+parsing process manager should
> either live in the Task SDK dist, or in a new separate dist that uses
> TaskSDK, but crucially not in apache-airflow-core. My reason for this is
> that I want it to be possible for the server components (scheduler, API
> server) to not need task-sdk installed (just for cleanliness/avoiding
> confusion about what versions it needs) and also vice-verse, to be able to
> run a “team worker bundle” (Dag parsing, workers, triggered/async workers)
> on whatever version of TaskSDK they choose, again without
> apache-airflow-core installed for avoidance of doubt.
>
> Generally I would like this as it means we can have a nicer separation of
> Core and Dag parsing code, as the dag parsing itself uses the SDK, it would
> be nice to have a proper server/client split, both from a tighter security
> point-of-view, but also from a code layout point of view.
>
> -ash
>
>
> > On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Well, you started it - so it's up to you to decide if you think we have
> > consensus, or whether we need a vote.
> >
> > And It's not a question of "informal" vote but it's rather clear
> following
> > the https://www.apache.org/foundation/voting.html that we either need a
> > LAZY CONSENSUS or VOTE thread. Both are formal.
> >
> > This is the difficult part when you have a proposal, to assess (by you)
> > whether we are converging to consensus or whether vote is needed. There
> is
> > no other body or "authority" to do it for you.
> >
> > J.
> >
> > On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <su...@gmail.com>
> > wrote:
> >
> >> Sorry for nudging again, but can we get into some consensus on this? I
> mean
> >> if this AIP isn't good enough, then we can drop it altogether and
> someone
> >> can rethink the whole thing. Should we do some kind of informal voting
> and
> >> close this thread?
> >>
> >> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >>>>> My main concern with this right now is the serialisation format of
> >> DAGs
> >>> —
> >>>> it wasn’t really designed with remote submission in mind, so it need
> >> some
> >>>> careful examination to see if it is fit for this purpose or not.
> >>>
> >>> I understand Ash's concerns - the format has not been designed with
> >>> size/speed optimization in mind so **possibly** we could design a
> >> different
> >>> format that would be better suited.
> >>>
> >>> BUT ... Done is better than perfect.
> >>>
> >>> I think there are a number of risks involved in changing the format and
> >> it
> >>> could significantly increase time of development with uncertain gains
> at
> >>> the end - also because of the progress in compression that happened
> over
> >>> the last few years.
> >>>
> >>> It might be a good idea to experiment a bit with different compression
> >>> algorithms for "our" dag representation and possibly we could find the
> >> best
> >>> algorithm for "airflow dag" type of json data. There are a lot of
> >>> repetitions in the JSON representation and I guess in "our" json
> >>> representation there are some artifacts and repeated sections that
> simply
> >>> might compress well with different algorithms. Also in this case
> >>> speed matters (and CPU trade-off).
> >>>
> >>> Looking at compression "theory" - before we experiment with it - there
> is
> >>> the relatively new standard "zstandard"
> https://github.com/facebook/zstd
> >>> compression opensourced in 2016 which I've heard good things about -
> >>> especially that it maintains a very good compression rate for text
> data,
> >>> but also it is tremendously fast - especially for decompression (which
> is
> >>> super important factor for us - we compress new DAG representation far
> >> less
> >>> often than decompress it in general case). It is standardized in RFC
> >>> https://datatracker.ietf.org/doc/html/rfc8878 and there are various
> >>> implementations and it is even being added to Python standard library
> in
> >>> Python 3.14 https://docs.python.org/3.14/library/compression.zstd.html
> >> and
> >>> there is a very well maintained python binding library
> >>> https://pypi.org/project/zstd/ to Yann Collet (algorithm author) ZSTD
> C
> >>> library. And libzstd is already part of our images - it is needed by
> >> other
> >>> dependencies of ours. All with BSD licence, directly usable by us.
> >>>
> >>> I think this one might be a good candidate for us to try, and possibly
> >> with
> >>> zstd we could achieve both size and CPU overhead that would be
> comparable
> >>> with any "new" format we could come up with - especially that we are
> >>> talking merely about processing a huge blob between "storable"
> >> (compressed)
> >>> and "locally usable" state (Python dict). We could likely use a
> streaming
> >>> JSON library (say the one that is used in Pydantic internally
> >>> https://github.com/pydantic/jiter - we already have it as part of
> >>> Pydantic)
> >>> to also save memory - we could stream decompressed stream into jitter
> so
> >>> that both the json dict and string representation does not have to be
> >>> loaded fully in memory at the same time. There are likely lots of
> >>> optimisations we could do - I mentioned possibly streaming the data
> from
> >>> API directly to DB (if this is possible - not sure)
> >>>
> >>> J.
> >>>
> >>>
> >>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <
> sumeet.manit@gmail.com>
> >>> wrote:
> >>>
> >>>>>
> >>>>> My main concern with this right now is the serialisation format of
> >>> DAGs —
> >>>>> it wasn’t really designed with remote submission in mind, so it need
> >>> some
> >>>>> careful examination to see if it is fit for this purpose or not.
> >>>>>
> >>>>
> >>>> I'm not sure on this point, cause if we are able to convert a DAG into
> >>>> JSON, then it has to be transferable over the internet.
> >>>>
> >>>> In particular One of the things I worry about is that the JSON can get
> >>> huge
> >>>>> — I’ve seem this as large as 10-20Mb for some dags
> >>>>
> >>>>
> >>>> Yeah, agree on this, thats why we can transfer compressed data instead
> >> of
> >>>> real json. Of course, this won't guarantee that the payload will
> always
> >>> be
> >>>> small enough, but we can't say that it'll definitely happen either.
> >>>>
> >>>> I also wonder if as part of this proposal we should move the Callback
> >>>>> requests off the dag parsers and on to the workers instead
> >>>>
> >>>> let's make such a "workfload" implementation stream that could support
> >>> both
> >>>>> - Deadlines and DAG parsing logic
> >>>>
> >>>>
> >>>> I don't have any strong opinion here, but it feels like it's gonna
> blow
> >>> up
> >>>> the scope of the AIP too much.
> >>>>
> >>>>
> >>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>>
> >>>>>> My main concern with this right now is the serialisation format of
> >>>> DAGs —
> >>>>> it wasn’t really designed with remote submission in mind, so it need
> >>> some
> >>>>> careful examination to see if it is fit for this purpose or not.
> >>>>>
> >>>>> Yep. That might be potentially a problem (or at least "need more
> >>>> resources
> >>>>> to run airflow") and that is where my "2x memory" came from if we do
> >> it
> >>>> in
> >>>>> a trivial way. Currently we a) keep the whole DAG in memory when
> >>>>> serializing it b) submit it to database (also using essentially some
> >>> kind
> >>>>> of API (implemented by the database client) - so we know the whole
> >>> thing
> >>>>> "might work" but indeed if you use a trivial implementation of
> >>> submitting
> >>>>> the whole json - it basically means that the whole json will have to
> >>> also
> >>>>> be kept in the memory of API server. But we also compress it when
> >>> needed
> >>>> -
> >>>>> I wonder what are the compression ratios we saw with those 10-20MBs
> >>> Dags
> >>>> -
> >>>>> if the problem is using strings where bool would suffice, compression
> >>>>> should generally help a lot. We could only ever send compressed data
> >>> over
> >>>>> the API - there seems to be no need to send "plain JSON" data over
> >> the
> >>>> API
> >>>>> or storing the plain JSON in the DB (of course that trades memory for
> >>>> CPU).
> >>>>>
> >>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
> >> support
> >>>> for
> >>>>> any kind if binary data streaming - because that could help a lot of
> >> if
> >>>> we
> >>>>> could use streaming HTTP API and chunk and append the binary chunks
> >>> (when
> >>>>> writing) - or read data in chunks ans stream them back via the API.
> >>> That
> >>>>> could seriously decrease the amount of memory needed by the API
> >> server
> >>> to
> >>>>> process such huge serialized dags.
> >>>>>
> >>>>> And yeah - I would also love the "execute task" to be implemented
> >> here
> >>> -
> >>>>> but I am not sure if this should be part of the same effort or maybe
> >> a
> >>>>> separate implementation? That sounds very loosely coupled with DB
> >>>>> isolation. And it seems a common theme - I think that would also make
> >>> the
> >>>>> sync Deadline alerts case that we discussed at the dev call today. I
> >>>> wonder
> >>>>> if that should not be kind of parallel (let's make such a "workfload"
> >>>>> implementation stream that could support both - Deadlines and DAG
> >>> parsing
> >>>>> logic. We have already two "users" for it and I really love the
> >> saying
> >>>> "if
> >>>>> you want to make something reusable - make it usable first" - seems
> >>> like
> >>>>> we might have good opportunity to make such workload implementation
> >>>> "doubly
> >>>>> used" from the beginning which would increase chances it will be
> >>>>> "reusable" for other things as well :).
> >>>>>
> >>>>> J.
> >>>>>
> >>>>>
> >>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> My main concern with this right now is the serialisation format of
> >>>> DAGs —
> >>>>>> it wasn’t really designed with remote submission in mind, so it
> >> need
> >>>> some
> >>>>>> careful examination to see if it is fit for this purpose or not.
> >>>>>>
> >>>>>> In particular One of the things I worry about is that the JSON can
> >>> get
> >>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which
> >> is
> >>>>>> likely due to things being included as text when a bool might
> >>> suffice,
> >>>>> for
> >>>>>> example) But I don’t think “just submit the existing JSON over an
> >>> API”
> >>>>> is a
> >>>>>> good idea.
> >>>>>>
> >>>>>> I also wonder if as part of this proposal we should move the
> >> Callback
> >>>>>> requests off the dag parsers and on to the workers instead — in
> >>> AIP-72
> >>>> we
> >>>>>> introduced the concept of a Workload, with the only one existing
> >>> right
> >>>>> now
> >>>>>> is “ExecuteTask”
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> >>>>>> — it might be time to finally move task and dag callbacks to the
> >> same
> >>>>> thing
> >>>>>> and make dag parsers only responsible for, well, parsing. :)
> >>>>>>
> >>>>>> These are all solvable problems, and this will be a great feature
> >> to
> >>>>> have,
> >>>>>> but we need to do some more thinking and planning first.
> >>>>>>
> >>>>>> -ash
> >>>>>>
> >>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> >> sumeet.manit@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Gentle reminder for everyone to review the proposal.
> >>>>>>>
> >>>>>>> Updated link:
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> >>>>>>>
> >>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> >>>>> sumeet.manit@gmail.com
> >>>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and others
> >>>>> suggested, I
> >>>>>>>> expanded the scope of this AIP and divided it into three phases.
> >>>> With
> >>>>>> the
> >>>>>>>> increased scope, the boundary line between this AIP and AIP-85
> >>> got a
> >>>>>> little
> >>>>>>>> thinner, but I believe these are still two different
> >> enhancements
> >>> to
> >>>>>> make.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> >>>>>> sumeet.manit@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Yeah, overall it makes sense to include Triggers as well to be
> >>> part
> >>>>> of
> >>>>>>>>> this AIP and phase out the implementation. Though I didn't
> >>> exclude
> >>>>>> Triggers
> >>>>>>>>> because "Uber" doesn't need that, I just thought of keeping the
> >>>> scope
> >>>>>> of
> >>>>>>>>> development small and achieving them, just like it was done in
> >>>>> Airlfow
> >>>>>> 3 by
> >>>>>>>>> secluding only workers and not DAG-processor & Triggers.
> >>>>>>>>>
> >>>>>>>>> But if you think Triggers should be part of this AIP itself,
> >>> then I
> >>>>> can
> >>>>>>>>> do that and include Triggers as well in it.
> >>>>>>>>>
> >>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <jarek@potiuk.com
> >>>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I would very much prefer the architectural choices of this AIP
> >>> are
> >>>>>> based
> >>>>>>>>>> on
> >>>>>>>>>> "general public" needs rather than "Uber needs" even if Uber
> >>> will
> >>>> be
> >>>>>>>>>> implementing it - so from my point of view having Trigger
> >>>> separation
> >>>>>> as
> >>>>>>>>>> part of it is quite important.
> >>>>>>>>>>
> >>>>>>>>>> But that's not even this.
> >>>>>>>>>>
> >>>>>>>>>> We've been discussing for example for Deadlines (being
> >>> implemented
> >>>>> by
> >>>>>>>>>> Dennis and Ramit a possibility of short, notification-style
> >>>>>> "deadlines"
> >>>>>>>>>> to be send to triggerer for execution - this is well advanced
> >>> now,
> >>>>> and
> >>>>>>>>>> whether you want it or not Dag-provided code might be
> >> serialized
> >>>> and
> >>>>>> sent
> >>>>>>>>>> to triggerer for execution. This is part of our "broader"
> >>>>>> architectural
> >>>>>>>>>> change where we treat "workers" and "triggerer" similarly as a
> >>>>> general
> >>>>>>>>>> executors of "sync" and "async" tasks respectively. That's
> >> where
> >>>>>> Airflow
> >>>>>>>>>> is
> >>>>>>>>>> evolving towards - inevitably.
> >>>>>>>>>>
> >>>>>>>>>> But we can of course phase things in out for implementation -
> >>> even
> >>>>> if
> >>>>>> AIP
> >>>>>>>>>> should cover both, I think if the goal of the AIP and preamble
> >>> is
> >>>>>> about
> >>>>>>>>>> separating "user code" from "database" as the main reason, it
> >>> also
> >>>>>> means
> >>>>>>>>>> Triggerer if you ask me (from design point of view at least).
> >>>>>>>>>>
> >>>>>>>>>> Again implementation can be phased and even different people
> >> and
> >>>>> teams
> >>>>>>>>>> might work on those phases/pieces.
> >>>>>>>>>>
> >>>>>>>>>> J.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> >>>>>> sumeet.manit@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers as
> >>>> well,
> >>>>>>>>>> but
> >>>>>>>>>>>> that
> >>>>>>>>>>>> can be done as part of a different AIP
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> You won't achieve your goal of "true" isolation of user code
> >> if
> >>>> you
> >>>>>>>>>> don't
> >>>>>>>>>>>> do triggerer. I think if the goal is to achieve it - it
> >> should
> >>>>> cover
> >>>>>>>>>>> both.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> My bad, should've explained our architecture for triggers as
> >>>> well,
> >>>>>>>>>>> apologies. So here it is:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> - Triggers would be running on a centralized service, so
> >> all
> >>>> the
> >>>>>>>>>> Trigger
> >>>>>>>>>>> classes will be part of the platform team's repo and not
> >> the
> >>>>>>>>>> customer's
> >>>>>>>>>>> repo
> >>>>>>>>>>> - The triggers won't be able to use any libs other than std
> >>>> ones,
> >>>>>>>>>> which
> >>>>>>>>>>> are being used in core Airflow (like requests, etc)
> >>>>>>>>>>> - As we are the owners of the core Airflow repo, customers
> >>> have
> >>>>> to
> >>>>>>>>>> get
> >>>>>>>>>>> our approval to land any class in this path (unlike the
> >> dags
> >>>> repo
> >>>>>>>>>> which
> >>>>>>>>>>> they own)
> >>>>>>>>>>> - When a customer's task defer, we would have an allowlist
> >> on
> >>>> our
> >>>>>>>>>> side
> >>>>>>>>>>> to check if we should do the async polling or not
> >>>>>>>>>>> - If the Trigger class isn't part of our repo (allowlist),
> >>> just
> >>>>>>>>>> fail the
> >>>>>>>>>>> task, as anyway we won't be having the code that they used
> >> in
> >>>> the
> >>>>>>>>>>> trigger
> >>>>>>>>>>> class
> >>>>>>>>>>> - If any of these conditions aren't suitable for you (as a
> >>>>>>>>>> customer),
> >>>>>>>>>>> feel free to use sync tasks only
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> But in general, I agree to make triggerer svc also
> >> communicate
> >>>> over
> >>>>>>>>>> apis
> >>>>>>>>>>> only. If that is done, then we can have instances of
> >> triggerer
> >>>> svc
> >>>>>>>>>> running
> >>>>>>>>>>> at customer's side as well, which can process any type of
> >>> trigger
> >>>>>>>>>> class.
> >>>>>>>>>>> Though that's not a blocker for us at the moment, cause
> >>> triggerer
> >>>>> are
> >>>>>>>>>>> mostly doing just polling using simple libs like requests.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> >>>>>>>>>> <ikholopov@google.com.invalid
> >>>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
> >>>>> aligns
> >>>>>>>>>> well
> >>>>>>>>>>>> with the goals of making Airflow well-scalable beyond a
> >>>>> single-team
> >>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full support
> >>> with
> >>>>> this
> >>>>>>>>>> one.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I've left a couple of clarification requests on the AIP
> >> page.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Igor
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> >>>>>>>>>>> sumeet.manit@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's good to
> >>> know
> >>>>>>>>>> that
> >>>>>>>>>>> the
> >>>>>>>>>>>>> DAG processor has some preemptive measures in place to
> >>> prevent
> >>>>>>>>>> access
> >>>>>>>>>>>>> to the DB. However, the main issue we are trying to solve
> >> is
> >>>> not
> >>>>> to
> >>>>>>>>>>>> provide
> >>>>>>>>>>>>> DB creds to the customer teams, who are using Airflow as a
> >>>>>>>>>> multi-tenant
> >>>>>>>>>>>>> orchestration platform. I've updated the doc to reflect
> >> this
> >>>>> point
> >>>>>>>>>> as
> >>>>>>>>>>>> well.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Answering Jarek's points,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism, added
> >>> that
> >>>>> in
> >>>>>>>>>> doc,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>> still how the token can be obtained (safely) is still open
> >> in
> >>>> my
> >>>>>>>>>> mind.
> >>>>>>>>>>> I
> >>>>>>>>>>>>> believe the token used by task executors can be created
> >>> outside
> >>>>> of
> >>>>>>>>>> it
> >>>>>>>>>>> as
> >>>>>>>>>>>>> well (I may be wrong here).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers as
> >>>> well,
> >>>>>>>>>> but
> >>>>>>>>>>>> that
> >>>>>>>>>>>>> can be done as part of a different AIP
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> #3. Yeah, I also believe the API should work largely.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can
> >>> work
> >>>>>>>>>> with
> >>>>>>>>>>>>> dag_bundles and every dag-processor instance would be
> >> treated
> >>>> as
> >>>>> a
> >>>>>>>>>> diff
> >>>>>>>>>>>>> bundle.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, added points around callbacks, as these are also
> >>> fetched
> >>>>>>>>>> directly
> >>>>>>>>>>>>> from the DB.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> >>>> jarek@potiuk.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
> >>> not
> >>>>>>>>>>>> protection
> >>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> >>> protect
> >>>>>>>>>> against
> >>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
> >> in
> >>>> the
> >>>>>>>>>>>> parsing
> >>>>>>>>>>>>>> process , as are the env var configs
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >>>>>>>>>>>>>> -
> >>>>>>>>>>>>>> is this perfect no? but it’s much more than no protection
> >>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed back
> >> then
> >>> in
> >>>>>>>>>> March
> >>>>>>>>>>> I
> >>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with full
> >>>> knowledge
> >>>>>>>>>> it's
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>> protecting against all threats.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> >>>>>>>>>> ash@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
> >>> not
> >>>>>>>>>>>> protection
> >>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
> >>> protect
> >>>>>>>>>>> against
> >>>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
> >>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>> parsing
> >>>>>>>>>>>>>>> process , as are the env var configs
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
> >> protection
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> >> jarek@potiuk.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Currently in the DagFile processor there is no built-in
> >>>>>>>>>>> protection
> >>>>>>>>>>>>>>> against
> >>>>>>>>>>>>>>>> user code from Dag Parsing to - for example - read
> >>> database
> >>>>>>>>>>>>>>>> credentials from airflow configuration and use them to
> >>> talk
> >>>>>>>>>> to DB
> >>>>>>>>>>>>>>> directly.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Ash Berlin-Taylor <as...@apache.org>.
This AIP is definitely heading in the right direction and is a feature I’d like to see.
For me the outstanding things that need more detail:
1. The Authentication token. How will this long lived token work without being insecure. Who and what will generate it? How will we identify top-level requests for Variables in order to be able to add Variable RBAC/ACLs. This is an important enough thing that I think it needs discussion before we vote on this AIP.
2. Security generally — how will this work, especially with the multi-team? I think this likely means making the APIs work on the bundle level as you mention in the doc, but I haven’t thought deeply about this yet.
3. API Versioning? One of the the key driving goals with AIP-72 and the Task Execution SDK was the idea that “you can upgrade the API server as you like, and your clients/workers never need to work” — i.e. the API server is 100% working with all older versions of the TaskSDK. I don’t know if we will achieve that goal in the long run but it is the desire, and part of why we are using CalVer and the Cadwyn library to provide API versioning.
4. As mentioned previously, not sure the existing serialised JSON format for DAGs is correct, but since that now has version and we already have the ability to upgrade that somewhere in the Airflow Core that doesn’t necessarily become a blocker/pre-requisite for this AIP.
I think Dag parsing API client+submission+parsing process manager should either live in the Task SDK dist, or in a new separate dist that uses TaskSDK, but crucially not in apache-airflow-core. My reason for this is that I want it to be possible for the server components (scheduler, API server) to not need task-sdk installed (just for cleanliness/avoiding confusion about what versions it needs) and also vice-verse, to be able to run a “team worker bundle” (Dag parsing, workers, triggered/async workers) on whatever version of TaskSDK they choose, again without apache-airflow-core installed for avoidance of doubt.
Generally I would like this as it means we can have a nicer separation of Core and Dag parsing code, as the dag parsing itself uses the SDK, it would be nice to have a proper server/client split, both from a tighter security point-of-view, but also from a code layout point of view.
-ash
> On 7 Aug 2025, at 12:36, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Well, you started it - so it's up to you to decide if you think we have
> consensus, or whether we need a vote.
>
> And It's not a question of "informal" vote but it's rather clear following
> the https://www.apache.org/foundation/voting.html that we either need a
> LAZY CONSENSUS or VOTE thread. Both are formal.
>
> This is the difficult part when you have a proposal, to assess (by you)
> whether we are converging to consensus or whether vote is needed. There is
> no other body or "authority" to do it for you.
>
> J.
>
> On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
>> Sorry for nudging again, but can we get into some consensus on this? I mean
>> if this AIP isn't good enough, then we can drop it altogether and someone
>> can rethink the whole thing. Should we do some kind of informal voting and
>> close this thread?
>>
>> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>>>> My main concern with this right now is the serialisation format of
>> DAGs
>>> —
>>>> it wasn’t really designed with remote submission in mind, so it need
>> some
>>>> careful examination to see if it is fit for this purpose or not.
>>>
>>> I understand Ash's concerns - the format has not been designed with
>>> size/speed optimization in mind so **possibly** we could design a
>> different
>>> format that would be better suited.
>>>
>>> BUT ... Done is better than perfect.
>>>
>>> I think there are a number of risks involved in changing the format and
>> it
>>> could significantly increase time of development with uncertain gains at
>>> the end - also because of the progress in compression that happened over
>>> the last few years.
>>>
>>> It might be a good idea to experiment a bit with different compression
>>> algorithms for "our" dag representation and possibly we could find the
>> best
>>> algorithm for "airflow dag" type of json data. There are a lot of
>>> repetitions in the JSON representation and I guess in "our" json
>>> representation there are some artifacts and repeated sections that simply
>>> might compress well with different algorithms. Also in this case
>>> speed matters (and CPU trade-off).
>>>
>>> Looking at compression "theory" - before we experiment with it - there is
>>> the relatively new standard "zstandard" https://github.com/facebook/zstd
>>> compression opensourced in 2016 which I've heard good things about -
>>> especially that it maintains a very good compression rate for text data,
>>> but also it is tremendously fast - especially for decompression (which is
>>> super important factor for us - we compress new DAG representation far
>> less
>>> often than decompress it in general case). It is standardized in RFC
>>> https://datatracker.ietf.org/doc/html/rfc8878 and there are various
>>> implementations and it is even being added to Python standard library in
>>> Python 3.14 https://docs.python.org/3.14/library/compression.zstd.html
>> and
>>> there is a very well maintained python binding library
>>> https://pypi.org/project/zstd/ to Yann Collet (algorithm author) ZSTD C
>>> library. And libzstd is already part of our images - it is needed by
>> other
>>> dependencies of ours. All with BSD licence, directly usable by us.
>>>
>>> I think this one might be a good candidate for us to try, and possibly
>> with
>>> zstd we could achieve both size and CPU overhead that would be comparable
>>> with any "new" format we could come up with - especially that we are
>>> talking merely about processing a huge blob between "storable"
>> (compressed)
>>> and "locally usable" state (Python dict). We could likely use a streaming
>>> JSON library (say the one that is used in Pydantic internally
>>> https://github.com/pydantic/jiter - we already have it as part of
>>> Pydantic)
>>> to also save memory - we could stream decompressed stream into jitter so
>>> that both the json dict and string representation does not have to be
>>> loaded fully in memory at the same time. There are likely lots of
>>> optimisations we could do - I mentioned possibly streaming the data from
>>> API directly to DB (if this is possible - not sure)
>>>
>>> J.
>>>
>>>
>>> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <su...@gmail.com>
>>> wrote:
>>>
>>>>>
>>>>> My main concern with this right now is the serialisation format of
>>> DAGs —
>>>>> it wasn’t really designed with remote submission in mind, so it need
>>> some
>>>>> careful examination to see if it is fit for this purpose or not.
>>>>>
>>>>
>>>> I'm not sure on this point, cause if we are able to convert a DAG into
>>>> JSON, then it has to be transferable over the internet.
>>>>
>>>> In particular One of the things I worry about is that the JSON can get
>>> huge
>>>>> — I’ve seem this as large as 10-20Mb for some dags
>>>>
>>>>
>>>> Yeah, agree on this, thats why we can transfer compressed data instead
>> of
>>>> real json. Of course, this won't guarantee that the payload will always
>>> be
>>>> small enough, but we can't say that it'll definitely happen either.
>>>>
>>>> I also wonder if as part of this proposal we should move the Callback
>>>>> requests off the dag parsers and on to the workers instead
>>>>
>>>> let's make such a "workfload" implementation stream that could support
>>> both
>>>>> - Deadlines and DAG parsing logic
>>>>
>>>>
>>>> I don't have any strong opinion here, but it feels like it's gonna blow
>>> up
>>>> the scope of the AIP too much.
>>>>
>>>>
>>>> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>>
>>>>>> My main concern with this right now is the serialisation format of
>>>> DAGs —
>>>>> it wasn’t really designed with remote submission in mind, so it need
>>> some
>>>>> careful examination to see if it is fit for this purpose or not.
>>>>>
>>>>> Yep. That might be potentially a problem (or at least "need more
>>>> resources
>>>>> to run airflow") and that is where my "2x memory" came from if we do
>> it
>>>> in
>>>>> a trivial way. Currently we a) keep the whole DAG in memory when
>>>>> serializing it b) submit it to database (also using essentially some
>>> kind
>>>>> of API (implemented by the database client) - so we know the whole
>>> thing
>>>>> "might work" but indeed if you use a trivial implementation of
>>> submitting
>>>>> the whole json - it basically means that the whole json will have to
>>> also
>>>>> be kept in the memory of API server. But we also compress it when
>>> needed
>>>> -
>>>>> I wonder what are the compression ratios we saw with those 10-20MBs
>>> Dags
>>>> -
>>>>> if the problem is using strings where bool would suffice, compression
>>>>> should generally help a lot. We could only ever send compressed data
>>> over
>>>>> the API - there seems to be no need to send "plain JSON" data over
>> the
>>>> API
>>>>> or storing the plain JSON in the DB (of course that trades memory for
>>>> CPU).
>>>>>
>>>>> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
>> support
>>>> for
>>>>> any kind if binary data streaming - because that could help a lot of
>> if
>>>> we
>>>>> could use streaming HTTP API and chunk and append the binary chunks
>>> (when
>>>>> writing) - or read data in chunks ans stream them back via the API.
>>> That
>>>>> could seriously decrease the amount of memory needed by the API
>> server
>>> to
>>>>> process such huge serialized dags.
>>>>>
>>>>> And yeah - I would also love the "execute task" to be implemented
>> here
>>> -
>>>>> but I am not sure if this should be part of the same effort or maybe
>> a
>>>>> separate implementation? That sounds very loosely coupled with DB
>>>>> isolation. And it seems a common theme - I think that would also make
>>> the
>>>>> sync Deadline alerts case that we discussed at the dev call today. I
>>>> wonder
>>>>> if that should not be kind of parallel (let's make such a "workfload"
>>>>> implementation stream that could support both - Deadlines and DAG
>>> parsing
>>>>> logic. We have already two "users" for it and I really love the
>> saying
>>>> "if
>>>>> you want to make something reusable - make it usable first" - seems
>>> like
>>>>> we might have good opportunity to make such workload implementation
>>>> "doubly
>>>>> used" from the beginning which would increase chances it will be
>>>>> "reusable" for other things as well :).
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org>
>>>> wrote:
>>>>>
>>>>>> My main concern with this right now is the serialisation format of
>>>> DAGs —
>>>>>> it wasn’t really designed with remote submission in mind, so it
>> need
>>>> some
>>>>>> careful examination to see if it is fit for this purpose or not.
>>>>>>
>>>>>> In particular One of the things I worry about is that the JSON can
>>> get
>>>>>> huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which
>> is
>>>>>> likely due to things being included as text when a bool might
>>> suffice,
>>>>> for
>>>>>> example) But I don’t think “just submit the existing JSON over an
>>> API”
>>>>> is a
>>>>>> good idea.
>>>>>>
>>>>>> I also wonder if as part of this proposal we should move the
>> Callback
>>>>>> requests off the dag parsers and on to the workers instead — in
>>> AIP-72
>>>> we
>>>>>> introduced the concept of a Workload, with the only one existing
>>> right
>>>>> now
>>>>>> is “ExecuteTask”
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
>>>>>> — it might be time to finally move task and dag callbacks to the
>> same
>>>>> thing
>>>>>> and make dag parsers only responsible for, well, parsing. :)
>>>>>>
>>>>>> These are all solvable problems, and this will be a great feature
>> to
>>>>> have,
>>>>>> but we need to do some more thinking and planning first.
>>>>>>
>>>>>> -ash
>>>>>>
>>>>>>> On 31 Jul 2025, at 10:12, Sumit Maheshwari <
>> sumeet.manit@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> Gentle reminder for everyone to review the proposal.
>>>>>>>
>>>>>>> Updated link:
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
>>>>>>>
>>>>>>> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
>>>>> sumeet.manit@gmail.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks everyone for reviewing this AIP. As Jarek and others
>>>>> suggested, I
>>>>>>>> expanded the scope of this AIP and divided it into three phases.
>>>> With
>>>>>> the
>>>>>>>> increased scope, the boundary line between this AIP and AIP-85
>>> got a
>>>>>> little
>>>>>>>> thinner, but I believe these are still two different
>> enhancements
>>> to
>>>>>> make.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
>>>>>> sumeet.manit@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yeah, overall it makes sense to include Triggers as well to be
>>> part
>>>>> of
>>>>>>>>> this AIP and phase out the implementation. Though I didn't
>>> exclude
>>>>>> Triggers
>>>>>>>>> because "Uber" doesn't need that, I just thought of keeping the
>>>> scope
>>>>>> of
>>>>>>>>> development small and achieving them, just like it was done in
>>>>> Airlfow
>>>>>> 3 by
>>>>>>>>> secluding only workers and not DAG-processor & Triggers.
>>>>>>>>>
>>>>>>>>> But if you think Triggers should be part of this AIP itself,
>>> then I
>>>>> can
>>>>>>>>> do that and include Triggers as well in it.
>>>>>>>>>
>>>>>>>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <jarek@potiuk.com
>>>
>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I would very much prefer the architectural choices of this AIP
>>> are
>>>>>> based
>>>>>>>>>> on
>>>>>>>>>> "general public" needs rather than "Uber needs" even if Uber
>>> will
>>>> be
>>>>>>>>>> implementing it - so from my point of view having Trigger
>>>> separation
>>>>>> as
>>>>>>>>>> part of it is quite important.
>>>>>>>>>>
>>>>>>>>>> But that's not even this.
>>>>>>>>>>
>>>>>>>>>> We've been discussing for example for Deadlines (being
>>> implemented
>>>>> by
>>>>>>>>>> Dennis and Ramit a possibility of short, notification-style
>>>>>> "deadlines"
>>>>>>>>>> to be send to triggerer for execution - this is well advanced
>>> now,
>>>>> and
>>>>>>>>>> whether you want it or not Dag-provided code might be
>> serialized
>>>> and
>>>>>> sent
>>>>>>>>>> to triggerer for execution. This is part of our "broader"
>>>>>> architectural
>>>>>>>>>> change where we treat "workers" and "triggerer" similarly as a
>>>>> general
>>>>>>>>>> executors of "sync" and "async" tasks respectively. That's
>> where
>>>>>> Airflow
>>>>>>>>>> is
>>>>>>>>>> evolving towards - inevitably.
>>>>>>>>>>
>>>>>>>>>> But we can of course phase things in out for implementation -
>>> even
>>>>> if
>>>>>> AIP
>>>>>>>>>> should cover both, I think if the goal of the AIP and preamble
>>> is
>>>>>> about
>>>>>>>>>> separating "user code" from "database" as the main reason, it
>>> also
>>>>>> means
>>>>>>>>>> Triggerer if you ask me (from design point of view at least).
>>>>>>>>>>
>>>>>>>>>> Again implementation can be phased and even different people
>> and
>>>>> teams
>>>>>>>>>> might work on those phases/pieces.
>>>>>>>>>>
>>>>>>>>>> J.
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
>>>>>> sumeet.manit@gmail.com
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers as
>>>> well,
>>>>>>>>>> but
>>>>>>>>>>>> that
>>>>>>>>>>>> can be done as part of a different AIP
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> You won't achieve your goal of "true" isolation of user code
>> if
>>>> you
>>>>>>>>>> don't
>>>>>>>>>>>> do triggerer. I think if the goal is to achieve it - it
>> should
>>>>> cover
>>>>>>>>>>> both.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> My bad, should've explained our architecture for triggers as
>>>> well,
>>>>>>>>>>> apologies. So here it is:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Triggers would be running on a centralized service, so
>> all
>>>> the
>>>>>>>>>> Trigger
>>>>>>>>>>> classes will be part of the platform team's repo and not
>> the
>>>>>>>>>> customer's
>>>>>>>>>>> repo
>>>>>>>>>>> - The triggers won't be able to use any libs other than std
>>>> ones,
>>>>>>>>>> which
>>>>>>>>>>> are being used in core Airflow (like requests, etc)
>>>>>>>>>>> - As we are the owners of the core Airflow repo, customers
>>> have
>>>>> to
>>>>>>>>>> get
>>>>>>>>>>> our approval to land any class in this path (unlike the
>> dags
>>>> repo
>>>>>>>>>> which
>>>>>>>>>>> they own)
>>>>>>>>>>> - When a customer's task defer, we would have an allowlist
>> on
>>>> our
>>>>>>>>>> side
>>>>>>>>>>> to check if we should do the async polling or not
>>>>>>>>>>> - If the Trigger class isn't part of our repo (allowlist),
>>> just
>>>>>>>>>> fail the
>>>>>>>>>>> task, as anyway we won't be having the code that they used
>> in
>>>> the
>>>>>>>>>>> trigger
>>>>>>>>>>> class
>>>>>>>>>>> - If any of these conditions aren't suitable for you (as a
>>>>>>>>>> customer),
>>>>>>>>>>> feel free to use sync tasks only
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> But in general, I agree to make triggerer svc also
>> communicate
>>>> over
>>>>>>>>>> apis
>>>>>>>>>>> only. If that is done, then we can have instances of
>> triggerer
>>>> svc
>>>>>>>>>> running
>>>>>>>>>>> at customer's side as well, which can process any type of
>>> trigger
>>>>>>>>>> class.
>>>>>>>>>>> Though that's not a blocker for us at the moment, cause
>>> triggerer
>>>>> are
>>>>>>>>>>> mostly doing just polling using simple libs like requests.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
>>>>>>>>>> <ikholopov@google.com.invalid
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
>>>>> aligns
>>>>>>>>>> well
>>>>>>>>>>>> with the goals of making Airflow well-scalable beyond a
>>>>> single-team
>>>>>>>>>>>> deployment (and AIP-85 goals), so you have my full support
>>> with
>>>>> this
>>>>>>>>>> one.
>>>>>>>>>>>>
>>>>>>>>>>>> I've left a couple of clarification requests on the AIP
>> page.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Igor
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
>>>>>>>>>>> sumeet.manit@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Jarek and Ash, for the initial review. It's good to
>>> know
>>>>>>>>>> that
>>>>>>>>>>> the
>>>>>>>>>>>>> DAG processor has some preemptive measures in place to
>>> prevent
>>>>>>>>>> access
>>>>>>>>>>>>> to the DB. However, the main issue we are trying to solve
>> is
>>>> not
>>>>> to
>>>>>>>>>>>> provide
>>>>>>>>>>>>> DB creds to the customer teams, who are using Airflow as a
>>>>>>>>>> multi-tenant
>>>>>>>>>>>>> orchestration platform. I've updated the doc to reflect
>> this
>>>>> point
>>>>>>>>>> as
>>>>>>>>>>>> well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Answering Jarek's points,
>>>>>>>>>>>>>
>>>>>>>>>>>>> #1. Yeah, had forgot to write about token mechanism, added
>>> that
>>>>> in
>>>>>>>>>> doc,
>>>>>>>>>>>> but
>>>>>>>>>>>>> still how the token can be obtained (safely) is still open
>> in
>>>> my
>>>>>>>>>> mind.
>>>>>>>>>>> I
>>>>>>>>>>>>> believe the token used by task executors can be created
>>> outside
>>>>> of
>>>>>>>>>> it
>>>>>>>>>>> as
>>>>>>>>>>>>> well (I may be wrong here).
>>>>>>>>>>>>>
>>>>>>>>>>>>> #2. Yeah, we would need something similar for triggerers as
>>>> well,
>>>>>>>>>> but
>>>>>>>>>>>> that
>>>>>>>>>>>>> can be done as part of a different AIP
>>>>>>>>>>>>>
>>>>>>>>>>>>> #3. Yeah, I also believe the API should work largely.
>>>>>>>>>>>>>
>>>>>>>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can
>>> work
>>>>>>>>>> with
>>>>>>>>>>>>> dag_bundles and every dag-processor instance would be
>> treated
>>>> as
>>>>> a
>>>>>>>>>> diff
>>>>>>>>>>>>> bundle.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, added points around callbacks, as these are also
>>> fetched
>>>>>>>>>> directly
>>>>>>>>>>>>> from the DB.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
>>>> jarek@potiuk.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
>>> not
>>>>>>>>>>>> protection
>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
>>> protect
>>>>>>>>>> against
>>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
>> in
>>>> the
>>>>>>>>>>>> parsing
>>>>>>>>>>>>>> process , as are the env var configs
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>> is this perfect no? but it’s much more than no protection
>>>>>>>>>>>>>> Oh absolutely.. This is exactly what we discussed back
>> then
>>> in
>>>>>>>>>> March
>>>>>>>>>>> I
>>>>>>>>>>>>>> think - and the way we decided to go for 3.0 with full
>>>> knowledge
>>>>>>>>>> it's
>>>>>>>>>>>> not
>>>>>>>>>>>>>> protecting against all threats.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
>>>>>>>>>> ash@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> A clarification to this - the dag parser today is likely
>>> not
>>>>>>>>>>>> protection
>>>>>>>>>>>>>>> against a dedicated malicious DAG author, but it does
>>> protect
>>>>>>>>>>> against
>>>>>>>>>>>>>>> casual DB access attempts - the db session is blanked out
>>> in
>>>>>>>>>> the
>>>>>>>>>>>>> parsing
>>>>>>>>>>>>>>> process , as are the env var configs
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>>>>>>>>>>>>>>> - is this perfect no? but it’s much more than no
>> protection
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
>> jarek@potiuk.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Currently in the DagFile processor there is no built-in
>>>>>>>>>>> protection
>>>>>>>>>>>>>>> against
>>>>>>>>>>>>>>>> user code from Dag Parsing to - for example - read
>>> database
>>>>>>>>>>>>>>>> credentials from airflow configuration and use them to
>>> talk
>>>>>>>>>> to DB
>>>>>>>>>>>>>>> directly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
Well, you started it - so it's up to you to decide if you think we have
consensus, or whether we need a vote.
And It's not a question of "informal" vote but it's rather clear following
the https://www.apache.org/foundation/voting.html that we either need a
LAZY CONSENSUS or VOTE thread. Both are formal.
This is the difficult part when you have a proposal, to assess (by you)
whether we are converging to consensus or whether vote is needed. There is
no other body or "authority" to do it for you.
J.
On Thu, Aug 7, 2025 at 1:02 PM Sumit Maheshwari <su...@gmail.com>
wrote:
> Sorry for nudging again, but can we get into some consensus on this? I mean
> if this AIP isn't good enough, then we can drop it altogether and someone
> can rethink the whole thing. Should we do some kind of informal voting and
> close this thread?
>
> On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > > My main concern with this right now is the serialisation format of
> DAGs
> > —
> > > it wasn’t really designed with remote submission in mind, so it need
> some
> > > careful examination to see if it is fit for this purpose or not.
> >
> > I understand Ash's concerns - the format has not been designed with
> > size/speed optimization in mind so **possibly** we could design a
> different
> > format that would be better suited.
> >
> > BUT ... Done is better than perfect.
> >
> > I think there are a number of risks involved in changing the format and
> it
> > could significantly increase time of development with uncertain gains at
> > the end - also because of the progress in compression that happened over
> > the last few years.
> >
> > It might be a good idea to experiment a bit with different compression
> > algorithms for "our" dag representation and possibly we could find the
> best
> > algorithm for "airflow dag" type of json data. There are a lot of
> > repetitions in the JSON representation and I guess in "our" json
> > representation there are some artifacts and repeated sections that simply
> > might compress well with different algorithms. Also in this case
> > speed matters (and CPU trade-off).
> >
> > Looking at compression "theory" - before we experiment with it - there is
> > the relatively new standard "zstandard" https://github.com/facebook/zstd
> > compression opensourced in 2016 which I've heard good things about -
> > especially that it maintains a very good compression rate for text data,
> > but also it is tremendously fast - especially for decompression (which is
> > super important factor for us - we compress new DAG representation far
> less
> > often than decompress it in general case). It is standardized in RFC
> > https://datatracker.ietf.org/doc/html/rfc8878 and there are various
> > implementations and it is even being added to Python standard library in
> > Python 3.14 https://docs.python.org/3.14/library/compression.zstd.html
> and
> > there is a very well maintained python binding library
> > https://pypi.org/project/zstd/ to Yann Collet (algorithm author) ZSTD C
> > library. And libzstd is already part of our images - it is needed by
> other
> > dependencies of ours. All with BSD licence, directly usable by us.
> >
> > I think this one might be a good candidate for us to try, and possibly
> with
> > zstd we could achieve both size and CPU overhead that would be comparable
> > with any "new" format we could come up with - especially that we are
> > talking merely about processing a huge blob between "storable"
> (compressed)
> > and "locally usable" state (Python dict). We could likely use a streaming
> > JSON library (say the one that is used in Pydantic internally
> > https://github.com/pydantic/jiter - we already have it as part of
> > Pydantic)
> > to also save memory - we could stream decompressed stream into jitter so
> > that both the json dict and string representation does not have to be
> > loaded fully in memory at the same time. There are likely lots of
> > optimisations we could do - I mentioned possibly streaming the data from
> > API directly to DB (if this is possible - not sure)
> >
> > J.
> >
> >
> > On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <su...@gmail.com>
> > wrote:
> >
> > > >
> > > > My main concern with this right now is the serialisation format of
> > DAGs —
> > > > it wasn’t really designed with remote submission in mind, so it need
> > some
> > > > careful examination to see if it is fit for this purpose or not.
> > > >
> > >
> > > I'm not sure on this point, cause if we are able to convert a DAG into
> > > JSON, then it has to be transferable over the internet.
> > >
> > > In particular One of the things I worry about is that the JSON can get
> > huge
> > > > — I’ve seem this as large as 10-20Mb for some dags
> > >
> > >
> > > Yeah, agree on this, thats why we can transfer compressed data instead
> of
> > > real json. Of course, this won't guarantee that the payload will always
> > be
> > > small enough, but we can't say that it'll definitely happen either.
> > >
> > > I also wonder if as part of this proposal we should move the Callback
> > > > requests off the dag parsers and on to the workers instead
> > >
> > > let's make such a "workfload" implementation stream that could support
> > both
> > > > - Deadlines and DAG parsing logic
> > >
> > >
> > > I don't have any strong opinion here, but it feels like it's gonna blow
> > up
> > > the scope of the AIP too much.
> > >
> > >
> > > On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > > My main concern with this right now is the serialisation format of
> > > DAGs —
> > > > it wasn’t really designed with remote submission in mind, so it need
> > some
> > > > careful examination to see if it is fit for this purpose or not.
> > > >
> > > > Yep. That might be potentially a problem (or at least "need more
> > > resources
> > > > to run airflow") and that is where my "2x memory" came from if we do
> it
> > > in
> > > > a trivial way. Currently we a) keep the whole DAG in memory when
> > > > serializing it b) submit it to database (also using essentially some
> > kind
> > > > of API (implemented by the database client) - so we know the whole
> > thing
> > > > "might work" but indeed if you use a trivial implementation of
> > submitting
> > > > the whole json - it basically means that the whole json will have to
> > also
> > > > be kept in the memory of API server. But we also compress it when
> > needed
> > > -
> > > > I wonder what are the compression ratios we saw with those 10-20MBs
> > Dags
> > > -
> > > > if the problem is using strings where bool would suffice, compression
> > > > should generally help a lot. We could only ever send compressed data
> > over
> > > > the API - there seems to be no need to send "plain JSON" data over
> the
> > > API
> > > > or storing the plain JSON in the DB (of course that trades memory for
> > > CPU).
> > > >
> > > > I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have
> support
> > > for
> > > > any kind if binary data streaming - because that could help a lot of
> if
> > > we
> > > > could use streaming HTTP API and chunk and append the binary chunks
> > (when
> > > > writing) - or read data in chunks ans stream them back via the API.
> > That
> > > > could seriously decrease the amount of memory needed by the API
> server
> > to
> > > > process such huge serialized dags.
> > > >
> > > > And yeah - I would also love the "execute task" to be implemented
> here
> > -
> > > > but I am not sure if this should be part of the same effort or maybe
> a
> > > > separate implementation? That sounds very loosely coupled with DB
> > > > isolation. And it seems a common theme - I think that would also make
> > the
> > > > sync Deadline alerts case that we discussed at the dev call today. I
> > > wonder
> > > > if that should not be kind of parallel (let's make such a "workfload"
> > > > implementation stream that could support both - Deadlines and DAG
> > parsing
> > > > logic. We have already two "users" for it and I really love the
> saying
> > > "if
> > > > you want to make something reusable - make it usable first" - seems
> > like
> > > > we might have good opportunity to make such workload implementation
> > > "doubly
> > > > used" from the beginning which would increase chances it will be
> > > > "reusable" for other things as well :).
> > > >
> > > > J.
> > > >
> > > >
> > > > On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > >
> > > > > My main concern with this right now is the serialisation format of
> > > DAGs —
> > > > > it wasn’t really designed with remote submission in mind, so it
> need
> > > some
> > > > > careful examination to see if it is fit for this purpose or not.
> > > > >
> > > > > In particular One of the things I worry about is that the JSON can
> > get
> > > > > huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which
> is
> > > > > likely due to things being included as text when a bool might
> > suffice,
> > > > for
> > > > > example) But I don’t think “just submit the existing JSON over an
> > API”
> > > > is a
> > > > > good idea.
> > > > >
> > > > > I also wonder if as part of this proposal we should move the
> Callback
> > > > > requests off the dag parsers and on to the workers instead — in
> > AIP-72
> > > we
> > > > > introduced the concept of a Workload, with the only one existing
> > right
> > > > now
> > > > > is “ExecuteTask”
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > > > > — it might be time to finally move task and dag callbacks to the
> same
> > > > thing
> > > > > and make dag parsers only responsible for, well, parsing. :)
> > > > >
> > > > > These are all solvable problems, and this will be a great feature
> to
> > > > have,
> > > > > but we need to do some more thinking and planning first.
> > > > >
> > > > > -ash
> > > > >
> > > > > > On 31 Jul 2025, at 10:12, Sumit Maheshwari <
> sumeet.manit@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Gentle reminder for everyone to review the proposal.
> > > > > >
> > > > > > Updated link:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > > > > >
> > > > > > On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > > > sumeet.manit@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Thanks everyone for reviewing this AIP. As Jarek and others
> > > > suggested, I
> > > > > >> expanded the scope of this AIP and divided it into three phases.
> > > With
> > > > > the
> > > > > >> increased scope, the boundary line between this AIP and AIP-85
> > got a
> > > > > little
> > > > > >> thinner, but I believe these are still two different
> enhancements
> > to
> > > > > make.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > > > sumeet.manit@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Yeah, overall it makes sense to include Triggers as well to be
> > part
> > > > of
> > > > > >>> this AIP and phase out the implementation. Though I didn't
> > exclude
> > > > > Triggers
> > > > > >>> because "Uber" doesn't need that, I just thought of keeping the
> > > scope
> > > > > of
> > > > > >>> development small and achieving them, just like it was done in
> > > > Airlfow
> > > > > 3 by
> > > > > >>> secluding only workers and not DAG-processor & Triggers.
> > > > > >>>
> > > > > >>> But if you think Triggers should be part of this AIP itself,
> > then I
> > > > can
> > > > > >>> do that and include Triggers as well in it.
> > > > > >>>
> > > > > >>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <jarek@potiuk.com
> >
> > > > wrote:
> > > > > >>>
> > > > > >>>> I would very much prefer the architectural choices of this AIP
> > are
> > > > > based
> > > > > >>>> on
> > > > > >>>> "general public" needs rather than "Uber needs" even if Uber
> > will
> > > be
> > > > > >>>> implementing it - so from my point of view having Trigger
> > > separation
> > > > > as
> > > > > >>>> part of it is quite important.
> > > > > >>>>
> > > > > >>>> But that's not even this.
> > > > > >>>>
> > > > > >>>> We've been discussing for example for Deadlines (being
> > implemented
> > > > by
> > > > > >>>> Dennis and Ramit a possibility of short, notification-style
> > > > > "deadlines"
> > > > > >>>> to be send to triggerer for execution - this is well advanced
> > now,
> > > > and
> > > > > >>>> whether you want it or not Dag-provided code might be
> serialized
> > > and
> > > > > sent
> > > > > >>>> to triggerer for execution. This is part of our "broader"
> > > > > architectural
> > > > > >>>> change where we treat "workers" and "triggerer" similarly as a
> > > > general
> > > > > >>>> executors of "sync" and "async" tasks respectively. That's
> where
> > > > > Airflow
> > > > > >>>> is
> > > > > >>>> evolving towards - inevitably.
> > > > > >>>>
> > > > > >>>> But we can of course phase things in out for implementation -
> > even
> > > > if
> > > > > AIP
> > > > > >>>> should cover both, I think if the goal of the AIP and preamble
> > is
> > > > > about
> > > > > >>>> separating "user code" from "database" as the main reason, it
> > also
> > > > > means
> > > > > >>>> Triggerer if you ask me (from design point of view at least).
> > > > > >>>>
> > > > > >>>> Again implementation can be phased and even different people
> and
> > > > teams
> > > > > >>>> might work on those phases/pieces.
> > > > > >>>>
> > > > > >>>> J.
> > > > > >>>>
> > > > > >>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > > > sumeet.manit@gmail.com
> > > > > >>>>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>>>
> > > > > >>>>>>> #2. Yeah, we would need something similar for triggerers as
> > > well,
> > > > > >>>> but
> > > > > >>>>>> that
> > > > > >>>>>> can be done as part of a different AIP
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> You won't achieve your goal of "true" isolation of user code
> if
> > > you
> > > > > >>>> don't
> > > > > >>>>>> do triggerer. I think if the goal is to achieve it - it
> should
> > > > cover
> > > > > >>>>> both.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> My bad, should've explained our architecture for triggers as
> > > well,
> > > > > >>>>> apologies. So here it is:
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> - Triggers would be running on a centralized service, so
> all
> > > the
> > > > > >>>> Trigger
> > > > > >>>>> classes will be part of the platform team's repo and not
> the
> > > > > >>>> customer's
> > > > > >>>>> repo
> > > > > >>>>> - The triggers won't be able to use any libs other than std
> > > ones,
> > > > > >>>> which
> > > > > >>>>> are being used in core Airflow (like requests, etc)
> > > > > >>>>> - As we are the owners of the core Airflow repo, customers
> > have
> > > > to
> > > > > >>>> get
> > > > > >>>>> our approval to land any class in this path (unlike the
> dags
> > > repo
> > > > > >>>> which
> > > > > >>>>> they own)
> > > > > >>>>> - When a customer's task defer, we would have an allowlist
> on
> > > our
> > > > > >>>> side
> > > > > >>>>> to check if we should do the async polling or not
> > > > > >>>>> - If the Trigger class isn't part of our repo (allowlist),
> > just
> > > > > >>>> fail the
> > > > > >>>>> task, as anyway we won't be having the code that they used
> in
> > > the
> > > > > >>>>> trigger
> > > > > >>>>> class
> > > > > >>>>> - If any of these conditions aren't suitable for you (as a
> > > > > >>>> customer),
> > > > > >>>>> feel free to use sync tasks only
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> But in general, I agree to make triggerer svc also
> communicate
> > > over
> > > > > >>>> apis
> > > > > >>>>> only. If that is done, then we can have instances of
> triggerer
> > > svc
> > > > > >>>> running
> > > > > >>>>> at customer's side as well, which can process any type of
> > trigger
> > > > > >>>> class.
> > > > > >>>>> Though that's not a blocker for us at the moment, cause
> > triggerer
> > > > are
> > > > > >>>>> mostly doing just polling using simple libs like requests.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > > > >>>> <ikholopov@google.com.invalid
> > > > > >>>>>>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
> > > > aligns
> > > > > >>>> well
> > > > > >>>>>> with the goals of making Airflow well-scalable beyond a
> > > > single-team
> > > > > >>>>>> deployment (and AIP-85 goals), so you have my full support
> > with
> > > > this
> > > > > >>>> one.
> > > > > >>>>>>
> > > > > >>>>>> I've left a couple of clarification requests on the AIP
> page.
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> Igor
> > > > > >>>>>>
> > > > > >>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > > > >>>>> sumeet.manit@gmail.com>
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Thanks Jarek and Ash, for the initial review. It's good to
> > know
> > > > > >>>> that
> > > > > >>>>> the
> > > > > >>>>>>> DAG processor has some preemptive measures in place to
> > prevent
> > > > > >>>> access
> > > > > >>>>>>> to the DB. However, the main issue we are trying to solve
> is
> > > not
> > > > to
> > > > > >>>>>> provide
> > > > > >>>>>>> DB creds to the customer teams, who are using Airflow as a
> > > > > >>>> multi-tenant
> > > > > >>>>>>> orchestration platform. I've updated the doc to reflect
> this
> > > > point
> > > > > >>>> as
> > > > > >>>>>> well.
> > > > > >>>>>>>
> > > > > >>>>>>> Answering Jarek's points,
> > > > > >>>>>>>
> > > > > >>>>>>> #1. Yeah, had forgot to write about token mechanism, added
> > that
> > > > in
> > > > > >>>> doc,
> > > > > >>>>>> but
> > > > > >>>>>>> still how the token can be obtained (safely) is still open
> in
> > > my
> > > > > >>>> mind.
> > > > > >>>>> I
> > > > > >>>>>>> believe the token used by task executors can be created
> > outside
> > > > of
> > > > > >>>> it
> > > > > >>>>> as
> > > > > >>>>>>> well (I may be wrong here).
> > > > > >>>>>>>
> > > > > >>>>>>> #2. Yeah, we would need something similar for triggerers as
> > > well,
> > > > > >>>> but
> > > > > >>>>>> that
> > > > > >>>>>>> can be done as part of a different AIP
> > > > > >>>>>>>
> > > > > >>>>>>> #3. Yeah, I also believe the API should work largely.
> > > > > >>>>>>>
> > > > > >>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can
> > work
> > > > > >>>> with
> > > > > >>>>>>> dag_bundles and every dag-processor instance would be
> treated
> > > as
> > > > a
> > > > > >>>> diff
> > > > > >>>>>>> bundle.
> > > > > >>>>>>>
> > > > > >>>>>>> Also, added points around callbacks, as these are also
> > fetched
> > > > > >>>> directly
> > > > > >>>>>>> from the DB.
> > > > > >>>>>>>
> > > > > >>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > > jarek@potiuk.com>
> > > > > >>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>>> A clarification to this - the dag parser today is likely
> > not
> > > > > >>>>>> protection
> > > > > >>>>>>>> against a dedicated malicious DAG author, but it does
> > protect
> > > > > >>>> against
> > > > > >>>>>>>> casual DB access attempts - the db session is blanked out
> in
> > > the
> > > > > >>>>>> parsing
> > > > > >>>>>>>> process , as are the env var configs
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > > >>>>>>>> -
> > > > > >>>>>>>> is this perfect no? but it’s much more than no protection
> > > > > >>>>>>>> Oh absolutely.. This is exactly what we discussed back
> then
> > in
> > > > > >>>> March
> > > > > >>>>> I
> > > > > >>>>>>>> think - and the way we decided to go for 3.0 with full
> > > knowledge
> > > > > >>>> it's
> > > > > >>>>>> not
> > > > > >>>>>>>> protecting against all threats.
> > > > > >>>>>>>>
> > > > > >>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > > > > >>>> ash@apache.org>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> A clarification to this - the dag parser today is likely
> > not
> > > > > >>>>>> protection
> > > > > >>>>>>>>> against a dedicated malicious DAG author, but it does
> > protect
> > > > > >>>>> against
> > > > > >>>>>>>>> casual DB access attempts - the db session is blanked out
> > in
> > > > > >>>> the
> > > > > >>>>>>> parsing
> > > > > >>>>>>>>> process , as are the env var configs
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > > >>>>>>>>> - is this perfect no? but it’s much more than no
> protection
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <
> jarek@potiuk.com>
> > > > > >>>> wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Currently in the DagFile processor there is no built-in
> > > > > >>>>> protection
> > > > > >>>>>>>>> against
> > > > > >>>>>>>>>> user code from Dag Parsing to - for example - read
> > database
> > > > > >>>>>>>>>> credentials from airflow configuration and use them to
> > talk
> > > > > >>>> to DB
> > > > > >>>>>>>>> directly.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Sorry for nudging again, but can we get into some consensus on this? I mean
if this AIP isn't good enough, then we can drop it altogether and someone
can rethink the whole thing. Should we do some kind of informal voting and
close this thread?
On Mon, Aug 4, 2025 at 3:32 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > > My main concern with this right now is the serialisation format of DAGs
> —
> > it wasn’t really designed with remote submission in mind, so it need some
> > careful examination to see if it is fit for this purpose or not.
>
> I understand Ash's concerns - the format has not been designed with
> size/speed optimization in mind so **possibly** we could design a different
> format that would be better suited.
>
> BUT ... Done is better than perfect.
>
> I think there are a number of risks involved in changing the format and it
> could significantly increase time of development with uncertain gains at
> the end - also because of the progress in compression that happened over
> the last few years.
>
> It might be a good idea to experiment a bit with different compression
> algorithms for "our" dag representation and possibly we could find the best
> algorithm for "airflow dag" type of json data. There are a lot of
> repetitions in the JSON representation and I guess in "our" json
> representation there are some artifacts and repeated sections that simply
> might compress well with different algorithms. Also in this case
> speed matters (and CPU trade-off).
>
> Looking at compression "theory" - before we experiment with it - there is
> the relatively new standard "zstandard" https://github.com/facebook/zstd
> compression opensourced in 2016 which I've heard good things about -
> especially that it maintains a very good compression rate for text data,
> but also it is tremendously fast - especially for decompression (which is
> super important factor for us - we compress new DAG representation far less
> often than decompress it in general case). It is standardized in RFC
> https://datatracker.ietf.org/doc/html/rfc8878 and there are various
> implementations and it is even being added to Python standard library in
> Python 3.14 https://docs.python.org/3.14/library/compression.zstd.html and
> there is a very well maintained python binding library
> https://pypi.org/project/zstd/ to Yann Collet (algorithm author) ZSTD C
> library. And libzstd is already part of our images - it is needed by other
> dependencies of ours. All with BSD licence, directly usable by us.
>
> I think this one might be a good candidate for us to try, and possibly with
> zstd we could achieve both size and CPU overhead that would be comparable
> with any "new" format we could come up with - especially that we are
> talking merely about processing a huge blob between "storable" (compressed)
> and "locally usable" state (Python dict). We could likely use a streaming
> JSON library (say the one that is used in Pydantic internally
> https://github.com/pydantic/jiter - we already have it as part of
> Pydantic)
> to also save memory - we could stream decompressed stream into jitter so
> that both the json dict and string representation does not have to be
> loaded fully in memory at the same time. There are likely lots of
> optimisations we could do - I mentioned possibly streaming the data from
> API directly to DB (if this is possible - not sure)
>
> J.
>
>
> On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
> > >
> > > My main concern with this right now is the serialisation format of
> DAGs —
> > > it wasn’t really designed with remote submission in mind, so it need
> some
> > > careful examination to see if it is fit for this purpose or not.
> > >
> >
> > I'm not sure on this point, cause if we are able to convert a DAG into
> > JSON, then it has to be transferable over the internet.
> >
> > In particular One of the things I worry about is that the JSON can get
> huge
> > > — I’ve seem this as large as 10-20Mb for some dags
> >
> >
> > Yeah, agree on this, thats why we can transfer compressed data instead of
> > real json. Of course, this won't guarantee that the payload will always
> be
> > small enough, but we can't say that it'll definitely happen either.
> >
> > I also wonder if as part of this proposal we should move the Callback
> > > requests off the dag parsers and on to the workers instead
> >
> > let's make such a "workfload" implementation stream that could support
> both
> > > - Deadlines and DAG parsing logic
> >
> >
> > I don't have any strong opinion here, but it feels like it's gonna blow
> up
> > the scope of the AIP too much.
> >
> >
> > On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > My main concern with this right now is the serialisation format of
> > DAGs —
> > > it wasn’t really designed with remote submission in mind, so it need
> some
> > > careful examination to see if it is fit for this purpose or not.
> > >
> > > Yep. That might be potentially a problem (or at least "need more
> > resources
> > > to run airflow") and that is where my "2x memory" came from if we do it
> > in
> > > a trivial way. Currently we a) keep the whole DAG in memory when
> > > serializing it b) submit it to database (also using essentially some
> kind
> > > of API (implemented by the database client) - so we know the whole
> thing
> > > "might work" but indeed if you use a trivial implementation of
> submitting
> > > the whole json - it basically means that the whole json will have to
> also
> > > be kept in the memory of API server. But we also compress it when
> needed
> > -
> > > I wonder what are the compression ratios we saw with those 10-20MBs
> Dags
> > -
> > > if the problem is using strings where bool would suffice, compression
> > > should generally help a lot. We could only ever send compressed data
> over
> > > the API - there seems to be no need to send "plain JSON" data over the
> > API
> > > or storing the plain JSON in the DB (of course that trades memory for
> > CPU).
> > >
> > > I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have support
> > for
> > > any kind if binary data streaming - because that could help a lot of if
> > we
> > > could use streaming HTTP API and chunk and append the binary chunks
> (when
> > > writing) - or read data in chunks ans stream them back via the API.
> That
> > > could seriously decrease the amount of memory needed by the API server
> to
> > > process such huge serialized dags.
> > >
> > > And yeah - I would also love the "execute task" to be implemented here
> -
> > > but I am not sure if this should be part of the same effort or maybe a
> > > separate implementation? That sounds very loosely coupled with DB
> > > isolation. And it seems a common theme - I think that would also make
> the
> > > sync Deadline alerts case that we discussed at the dev call today. I
> > wonder
> > > if that should not be kind of parallel (let's make such a "workfload"
> > > implementation stream that could support both - Deadlines and DAG
> parsing
> > > logic. We have already two "users" for it and I really love the saying
> > "if
> > > you want to make something reusable - make it usable first" - seems
> like
> > > we might have good opportunity to make such workload implementation
> > "doubly
> > > used" from the beginning which would increase chances it will be
> > > "reusable" for other things as well :).
> > >
> > > J.
> > >
> > >
> > > On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > > > My main concern with this right now is the serialisation format of
> > DAGs —
> > > > it wasn’t really designed with remote submission in mind, so it need
> > some
> > > > careful examination to see if it is fit for this purpose or not.
> > > >
> > > > In particular One of the things I worry about is that the JSON can
> get
> > > > huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which is
> > > > likely due to things being included as text when a bool might
> suffice,
> > > for
> > > > example) But I don’t think “just submit the existing JSON over an
> API”
> > > is a
> > > > good idea.
> > > >
> > > > I also wonder if as part of this proposal we should move the Callback
> > > > requests off the dag parsers and on to the workers instead — in
> AIP-72
> > we
> > > > introduced the concept of a Workload, with the only one existing
> right
> > > now
> > > > is “ExecuteTask”
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > > > — it might be time to finally move task and dag callbacks to the same
> > > thing
> > > > and make dag parsers only responsible for, well, parsing. :)
> > > >
> > > > These are all solvable problems, and this will be a great feature to
> > > have,
> > > > but we need to do some more thinking and planning first.
> > > >
> > > > -ash
> > > >
> > > > > On 31 Jul 2025, at 10:12, Sumit Maheshwari <sumeet.manit@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Gentle reminder for everyone to review the proposal.
> > > > >
> > > > > Updated link:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > > > >
> > > > > On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > > sumeet.manit@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Thanks everyone for reviewing this AIP. As Jarek and others
> > > suggested, I
> > > > >> expanded the scope of this AIP and divided it into three phases.
> > With
> > > > the
> > > > >> increased scope, the boundary line between this AIP and AIP-85
> got a
> > > > little
> > > > >> thinner, but I believe these are still two different enhancements
> to
> > > > make.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > > sumeet.manit@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Yeah, overall it makes sense to include Triggers as well to be
> part
> > > of
> > > > >>> this AIP and phase out the implementation. Though I didn't
> exclude
> > > > Triggers
> > > > >>> because "Uber" doesn't need that, I just thought of keeping the
> > scope
> > > > of
> > > > >>> development small and achieving them, just like it was done in
> > > Airlfow
> > > > 3 by
> > > > >>> secluding only workers and not DAG-processor & Triggers.
> > > > >>>
> > > > >>> But if you think Triggers should be part of this AIP itself,
> then I
> > > can
> > > > >>> do that and include Triggers as well in it.
> > > > >>>
> > > > >>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > >>>
> > > > >>>> I would very much prefer the architectural choices of this AIP
> are
> > > > based
> > > > >>>> on
> > > > >>>> "general public" needs rather than "Uber needs" even if Uber
> will
> > be
> > > > >>>> implementing it - so from my point of view having Trigger
> > separation
> > > > as
> > > > >>>> part of it is quite important.
> > > > >>>>
> > > > >>>> But that's not even this.
> > > > >>>>
> > > > >>>> We've been discussing for example for Deadlines (being
> implemented
> > > by
> > > > >>>> Dennis and Ramit a possibility of short, notification-style
> > > > "deadlines"
> > > > >>>> to be send to triggerer for execution - this is well advanced
> now,
> > > and
> > > > >>>> whether you want it or not Dag-provided code might be serialized
> > and
> > > > sent
> > > > >>>> to triggerer for execution. This is part of our "broader"
> > > > architectural
> > > > >>>> change where we treat "workers" and "triggerer" similarly as a
> > > general
> > > > >>>> executors of "sync" and "async" tasks respectively. That's where
> > > > Airflow
> > > > >>>> is
> > > > >>>> evolving towards - inevitably.
> > > > >>>>
> > > > >>>> But we can of course phase things in out for implementation -
> even
> > > if
> > > > AIP
> > > > >>>> should cover both, I think if the goal of the AIP and preamble
> is
> > > > about
> > > > >>>> separating "user code" from "database" as the main reason, it
> also
> > > > means
> > > > >>>> Triggerer if you ask me (from design point of view at least).
> > > > >>>>
> > > > >>>> Again implementation can be phased and even different people and
> > > teams
> > > > >>>> might work on those phases/pieces.
> > > > >>>>
> > > > >>>> J.
> > > > >>>>
> > > > >>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > > sumeet.manit@gmail.com
> > > > >>>>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>>>
> > > > >>>>>>> #2. Yeah, we would need something similar for triggerers as
> > well,
> > > > >>>> but
> > > > >>>>>> that
> > > > >>>>>> can be done as part of a different AIP
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> You won't achieve your goal of "true" isolation of user code if
> > you
> > > > >>>> don't
> > > > >>>>>> do triggerer. I think if the goal is to achieve it - it should
> > > cover
> > > > >>>>> both.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> My bad, should've explained our architecture for triggers as
> > well,
> > > > >>>>> apologies. So here it is:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> - Triggers would be running on a centralized service, so all
> > the
> > > > >>>> Trigger
> > > > >>>>> classes will be part of the platform team's repo and not the
> > > > >>>> customer's
> > > > >>>>> repo
> > > > >>>>> - The triggers won't be able to use any libs other than std
> > ones,
> > > > >>>> which
> > > > >>>>> are being used in core Airflow (like requests, etc)
> > > > >>>>> - As we are the owners of the core Airflow repo, customers
> have
> > > to
> > > > >>>> get
> > > > >>>>> our approval to land any class in this path (unlike the dags
> > repo
> > > > >>>> which
> > > > >>>>> they own)
> > > > >>>>> - When a customer's task defer, we would have an allowlist on
> > our
> > > > >>>> side
> > > > >>>>> to check if we should do the async polling or not
> > > > >>>>> - If the Trigger class isn't part of our repo (allowlist),
> just
> > > > >>>> fail the
> > > > >>>>> task, as anyway we won't be having the code that they used in
> > the
> > > > >>>>> trigger
> > > > >>>>> class
> > > > >>>>> - If any of these conditions aren't suitable for you (as a
> > > > >>>> customer),
> > > > >>>>> feel free to use sync tasks only
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> But in general, I agree to make triggerer svc also communicate
> > over
> > > > >>>> apis
> > > > >>>>> only. If that is done, then we can have instances of triggerer
> > svc
> > > > >>>> running
> > > > >>>>> at customer's side as well, which can process any type of
> trigger
> > > > >>>> class.
> > > > >>>>> Though that's not a blocker for us at the moment, cause
> triggerer
> > > are
> > > > >>>>> mostly doing just polling using simple libs like requests.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > > >>>> <ikholopov@google.com.invalid
> > > > >>>>>>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
> > > aligns
> > > > >>>> well
> > > > >>>>>> with the goals of making Airflow well-scalable beyond a
> > > single-team
> > > > >>>>>> deployment (and AIP-85 goals), so you have my full support
> with
> > > this
> > > > >>>> one.
> > > > >>>>>>
> > > > >>>>>> I've left a couple of clarification requests on the AIP page.
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> Igor
> > > > >>>>>>
> > > > >>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > > >>>>> sumeet.manit@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Thanks Jarek and Ash, for the initial review. It's good to
> know
> > > > >>>> that
> > > > >>>>> the
> > > > >>>>>>> DAG processor has some preemptive measures in place to
> prevent
> > > > >>>> access
> > > > >>>>>>> to the DB. However, the main issue we are trying to solve is
> > not
> > > to
> > > > >>>>>> provide
> > > > >>>>>>> DB creds to the customer teams, who are using Airflow as a
> > > > >>>> multi-tenant
> > > > >>>>>>> orchestration platform. I've updated the doc to reflect this
> > > point
> > > > >>>> as
> > > > >>>>>> well.
> > > > >>>>>>>
> > > > >>>>>>> Answering Jarek's points,
> > > > >>>>>>>
> > > > >>>>>>> #1. Yeah, had forgot to write about token mechanism, added
> that
> > > in
> > > > >>>> doc,
> > > > >>>>>> but
> > > > >>>>>>> still how the token can be obtained (safely) is still open in
> > my
> > > > >>>> mind.
> > > > >>>>> I
> > > > >>>>>>> believe the token used by task executors can be created
> outside
> > > of
> > > > >>>> it
> > > > >>>>> as
> > > > >>>>>>> well (I may be wrong here).
> > > > >>>>>>>
> > > > >>>>>>> #2. Yeah, we would need something similar for triggerers as
> > well,
> > > > >>>> but
> > > > >>>>>> that
> > > > >>>>>>> can be done as part of a different AIP
> > > > >>>>>>>
> > > > >>>>>>> #3. Yeah, I also believe the API should work largely.
> > > > >>>>>>>
> > > > >>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can
> work
> > > > >>>> with
> > > > >>>>>>> dag_bundles and every dag-processor instance would be treated
> > as
> > > a
> > > > >>>> diff
> > > > >>>>>>> bundle.
> > > > >>>>>>>
> > > > >>>>>>> Also, added points around callbacks, as these are also
> fetched
> > > > >>>> directly
> > > > >>>>>>> from the DB.
> > > > >>>>>>>
> > > > >>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> > jarek@potiuk.com>
> > > > >>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>>> A clarification to this - the dag parser today is likely
> not
> > > > >>>>>> protection
> > > > >>>>>>>> against a dedicated malicious DAG author, but it does
> protect
> > > > >>>> against
> > > > >>>>>>>> casual DB access attempts - the db session is blanked out in
> > the
> > > > >>>>>> parsing
> > > > >>>>>>>> process , as are the env var configs
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > >>>>>>>> -
> > > > >>>>>>>> is this perfect no? but it’s much more than no protection
> > > > >>>>>>>> Oh absolutely.. This is exactly what we discussed back then
> in
> > > > >>>> March
> > > > >>>>> I
> > > > >>>>>>>> think - and the way we decided to go for 3.0 with full
> > knowledge
> > > > >>>> it's
> > > > >>>>>> not
> > > > >>>>>>>> protecting against all threats.
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > > > >>>> ash@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> A clarification to this - the dag parser today is likely
> not
> > > > >>>>>> protection
> > > > >>>>>>>>> against a dedicated malicious DAG author, but it does
> protect
> > > > >>>>> against
> > > > >>>>>>>>> casual DB access attempts - the db session is blanked out
> in
> > > > >>>> the
> > > > >>>>>>> parsing
> > > > >>>>>>>>> process , as are the env var configs
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > >>>>>>>>> - is this perfect no? but it’s much more than no protection
> > > > >>>>>>>>>
> > > > >>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
> > > > >>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Currently in the DagFile processor there is no built-in
> > > > >>>>> protection
> > > > >>>>>>>>> against
> > > > >>>>>>>>>> user code from Dag Parsing to - for example - read
> database
> > > > >>>>>>>>>> credentials from airflow configuration and use them to
> talk
> > > > >>>> to DB
> > > > >>>>>>>>> directly.
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
> > My main concern with this right now is the serialisation format of DAGs
—
> it wasn’t really designed with remote submission in mind, so it need some
> careful examination to see if it is fit for this purpose or not.
I understand Ash's concerns - the format has not been designed with
size/speed optimization in mind so **possibly** we could design a different
format that would be better suited.
BUT ... Done is better than perfect.
I think there are a number of risks involved in changing the format and it
could significantly increase time of development with uncertain gains at
the end - also because of the progress in compression that happened over
the last few years.
It might be a good idea to experiment a bit with different compression
algorithms for "our" dag representation and possibly we could find the best
algorithm for "airflow dag" type of json data. There are a lot of
repetitions in the JSON representation and I guess in "our" json
representation there are some artifacts and repeated sections that simply
might compress well with different algorithms. Also in this case
speed matters (and CPU trade-off).
Looking at compression "theory" - before we experiment with it - there is
the relatively new standard "zstandard" https://github.com/facebook/zstd
compression opensourced in 2016 which I've heard good things about -
especially that it maintains a very good compression rate for text data,
but also it is tremendously fast - especially for decompression (which is
super important factor for us - we compress new DAG representation far less
often than decompress it in general case). It is standardized in RFC
https://datatracker.ietf.org/doc/html/rfc8878 and there are various
implementations and it is even being added to Python standard library in
Python 3.14 https://docs.python.org/3.14/library/compression.zstd.html and
there is a very well maintained python binding library
https://pypi.org/project/zstd/ to Yann Collet (algorithm author) ZSTD C
library. And libzstd is already part of our images - it is needed by other
dependencies of ours. All with BSD licence, directly usable by us.
I think this one might be a good candidate for us to try, and possibly with
zstd we could achieve both size and CPU overhead that would be comparable
with any "new" format we could come up with - especially that we are
talking merely about processing a huge blob between "storable" (compressed)
and "locally usable" state (Python dict). We could likely use a streaming
JSON library (say the one that is used in Pydantic internally
https://github.com/pydantic/jiter - we already have it as part of Pydantic)
to also save memory - we could stream decompressed stream into jitter so
that both the json dict and string representation does not have to be
loaded fully in memory at the same time. There are likely lots of
optimisations we could do - I mentioned possibly streaming the data from
API directly to DB (if this is possible - not sure)
J.
On Mon, Aug 4, 2025 at 9:10 AM Sumit Maheshwari <su...@gmail.com>
wrote:
> >
> > My main concern with this right now is the serialisation format of DAGs —
> > it wasn’t really designed with remote submission in mind, so it need some
> > careful examination to see if it is fit for this purpose or not.
> >
>
> I'm not sure on this point, cause if we are able to convert a DAG into
> JSON, then it has to be transferable over the internet.
>
> In particular One of the things I worry about is that the JSON can get huge
> > — I’ve seem this as large as 10-20Mb for some dags
>
>
> Yeah, agree on this, thats why we can transfer compressed data instead of
> real json. Of course, this won't guarantee that the payload will always be
> small enough, but we can't say that it'll definitely happen either.
>
> I also wonder if as part of this proposal we should move the Callback
> > requests off the dag parsers and on to the workers instead
>
> let's make such a "workfload" implementation stream that could support both
> > - Deadlines and DAG parsing logic
>
>
> I don't have any strong opinion here, but it feels like it's gonna blow up
> the scope of the AIP too much.
>
>
> On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > My main concern with this right now is the serialisation format of
> DAGs —
> > it wasn’t really designed with remote submission in mind, so it need some
> > careful examination to see if it is fit for this purpose or not.
> >
> > Yep. That might be potentially a problem (or at least "need more
> resources
> > to run airflow") and that is where my "2x memory" came from if we do it
> in
> > a trivial way. Currently we a) keep the whole DAG in memory when
> > serializing it b) submit it to database (also using essentially some kind
> > of API (implemented by the database client) - so we know the whole thing
> > "might work" but indeed if you use a trivial implementation of submitting
> > the whole json - it basically means that the whole json will have to also
> > be kept in the memory of API server. But we also compress it when needed
> -
> > I wonder what are the compression ratios we saw with those 10-20MBs Dags
> -
> > if the problem is using strings where bool would suffice, compression
> > should generally help a lot. We could only ever send compressed data over
> > the API - there seems to be no need to send "plain JSON" data over the
> API
> > or storing the plain JSON in the DB (of course that trades memory for
> CPU).
> >
> > I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have support
> for
> > any kind if binary data streaming - because that could help a lot of if
> we
> > could use streaming HTTP API and chunk and append the binary chunks (when
> > writing) - or read data in chunks ans stream them back via the API. That
> > could seriously decrease the amount of memory needed by the API server to
> > process such huge serialized dags.
> >
> > And yeah - I would also love the "execute task" to be implemented here -
> > but I am not sure if this should be part of the same effort or maybe a
> > separate implementation? That sounds very loosely coupled with DB
> > isolation. And it seems a common theme - I think that would also make the
> > sync Deadline alerts case that we discussed at the dev call today. I
> wonder
> > if that should not be kind of parallel (let's make such a "workfload"
> > implementation stream that could support both - Deadlines and DAG parsing
> > logic. We have already two "users" for it and I really love the saying
> "if
> > you want to make something reusable - make it usable first" - seems like
> > we might have good opportunity to make such workload implementation
> "doubly
> > used" from the beginning which would increase chances it will be
> > "reusable" for other things as well :).
> >
> > J.
> >
> >
> > On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > My main concern with this right now is the serialisation format of
> DAGs —
> > > it wasn’t really designed with remote submission in mind, so it need
> some
> > > careful examination to see if it is fit for this purpose or not.
> > >
> > > In particular One of the things I worry about is that the JSON can get
> > > huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which is
> > > likely due to things being included as text when a bool might suffice,
> > for
> > > example) But I don’t think “just submit the existing JSON over an API”
> > is a
> > > good idea.
> > >
> > > I also wonder if as part of this proposal we should move the Callback
> > > requests off the dag parsers and on to the workers instead — in AIP-72
> we
> > > introduced the concept of a Workload, with the only one existing right
> > now
> > > is “ExecuteTask”
> > >
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > > — it might be time to finally move task and dag callbacks to the same
> > thing
> > > and make dag parsers only responsible for, well, parsing. :)
> > >
> > > These are all solvable problems, and this will be a great feature to
> > have,
> > > but we need to do some more thinking and planning first.
> > >
> > > -ash
> > >
> > > > On 31 Jul 2025, at 10:12, Sumit Maheshwari <su...@gmail.com>
> > > wrote:
> > > >
> > > > Gentle reminder for everyone to review the proposal.
> > > >
> > > > Updated link:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > > >
> > > > On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> > sumeet.manit@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Thanks everyone for reviewing this AIP. As Jarek and others
> > suggested, I
> > > >> expanded the scope of this AIP and divided it into three phases.
> With
> > > the
> > > >> increased scope, the boundary line between this AIP and AIP-85 got a
> > > little
> > > >> thinner, but I believe these are still two different enhancements to
> > > make.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > > sumeet.manit@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Yeah, overall it makes sense to include Triggers as well to be part
> > of
> > > >>> this AIP and phase out the implementation. Though I didn't exclude
> > > Triggers
> > > >>> because "Uber" doesn't need that, I just thought of keeping the
> scope
> > > of
> > > >>> development small and achieving them, just like it was done in
> > Airlfow
> > > 3 by
> > > >>> secluding only workers and not DAG-processor & Triggers.
> > > >>>
> > > >>> But if you think Triggers should be part of this AIP itself, then I
> > can
> > > >>> do that and include Triggers as well in it.
> > > >>>
> > > >>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >>>
> > > >>>> I would very much prefer the architectural choices of this AIP are
> > > based
> > > >>>> on
> > > >>>> "general public" needs rather than "Uber needs" even if Uber will
> be
> > > >>>> implementing it - so from my point of view having Trigger
> separation
> > > as
> > > >>>> part of it is quite important.
> > > >>>>
> > > >>>> But that's not even this.
> > > >>>>
> > > >>>> We've been discussing for example for Deadlines (being implemented
> > by
> > > >>>> Dennis and Ramit a possibility of short, notification-style
> > > "deadlines"
> > > >>>> to be send to triggerer for execution - this is well advanced now,
> > and
> > > >>>> whether you want it or not Dag-provided code might be serialized
> and
> > > sent
> > > >>>> to triggerer for execution. This is part of our "broader"
> > > architectural
> > > >>>> change where we treat "workers" and "triggerer" similarly as a
> > general
> > > >>>> executors of "sync" and "async" tasks respectively. That's where
> > > Airflow
> > > >>>> is
> > > >>>> evolving towards - inevitably.
> > > >>>>
> > > >>>> But we can of course phase things in out for implementation - even
> > if
> > > AIP
> > > >>>> should cover both, I think if the goal of the AIP and preamble is
> > > about
> > > >>>> separating "user code" from "database" as the main reason, it also
> > > means
> > > >>>> Triggerer if you ask me (from design point of view at least).
> > > >>>>
> > > >>>> Again implementation can be phased and even different people and
> > teams
> > > >>>> might work on those phases/pieces.
> > > >>>>
> > > >>>> J.
> > > >>>>
> > > >>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > > sumeet.manit@gmail.com
> > > >>>>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>>>
> > > >>>>>>> #2. Yeah, we would need something similar for triggerers as
> well,
> > > >>>> but
> > > >>>>>> that
> > > >>>>>> can be done as part of a different AIP
> > > >>>>>
> > > >>>>>
> > > >>>>> You won't achieve your goal of "true" isolation of user code if
> you
> > > >>>> don't
> > > >>>>>> do triggerer. I think if the goal is to achieve it - it should
> > cover
> > > >>>>> both.
> > > >>>>>
> > > >>>>>
> > > >>>>> My bad, should've explained our architecture for triggers as
> well,
> > > >>>>> apologies. So here it is:
> > > >>>>>
> > > >>>>>
> > > >>>>> - Triggers would be running on a centralized service, so all
> the
> > > >>>> Trigger
> > > >>>>> classes will be part of the platform team's repo and not the
> > > >>>> customer's
> > > >>>>> repo
> > > >>>>> - The triggers won't be able to use any libs other than std
> ones,
> > > >>>> which
> > > >>>>> are being used in core Airflow (like requests, etc)
> > > >>>>> - As we are the owners of the core Airflow repo, customers have
> > to
> > > >>>> get
> > > >>>>> our approval to land any class in this path (unlike the dags
> repo
> > > >>>> which
> > > >>>>> they own)
> > > >>>>> - When a customer's task defer, we would have an allowlist on
> our
> > > >>>> side
> > > >>>>> to check if we should do the async polling or not
> > > >>>>> - If the Trigger class isn't part of our repo (allowlist), just
> > > >>>> fail the
> > > >>>>> task, as anyway we won't be having the code that they used in
> the
> > > >>>>> trigger
> > > >>>>> class
> > > >>>>> - If any of these conditions aren't suitable for you (as a
> > > >>>> customer),
> > > >>>>> feel free to use sync tasks only
> > > >>>>>
> > > >>>>>
> > > >>>>> But in general, I agree to make triggerer svc also communicate
> over
> > > >>>> apis
> > > >>>>> only. If that is done, then we can have instances of triggerer
> svc
> > > >>>> running
> > > >>>>> at customer's side as well, which can process any type of trigger
> > > >>>> class.
> > > >>>>> Though that's not a blocker for us at the moment, cause triggerer
> > are
> > > >>>>> mostly doing just polling using simple libs like requests.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > > >>>> <ikholopov@google.com.invalid
> > > >>>>>>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
> > aligns
> > > >>>> well
> > > >>>>>> with the goals of making Airflow well-scalable beyond a
> > single-team
> > > >>>>>> deployment (and AIP-85 goals), so you have my full support with
> > this
> > > >>>> one.
> > > >>>>>>
> > > >>>>>> I've left a couple of clarification requests on the AIP page.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Igor
> > > >>>>>>
> > > >>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > > >>>>> sumeet.manit@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks Jarek and Ash, for the initial review. It's good to know
> > > >>>> that
> > > >>>>> the
> > > >>>>>>> DAG processor has some preemptive measures in place to prevent
> > > >>>> access
> > > >>>>>>> to the DB. However, the main issue we are trying to solve is
> not
> > to
> > > >>>>>> provide
> > > >>>>>>> DB creds to the customer teams, who are using Airflow as a
> > > >>>> multi-tenant
> > > >>>>>>> orchestration platform. I've updated the doc to reflect this
> > point
> > > >>>> as
> > > >>>>>> well.
> > > >>>>>>>
> > > >>>>>>> Answering Jarek's points,
> > > >>>>>>>
> > > >>>>>>> #1. Yeah, had forgot to write about token mechanism, added that
> > in
> > > >>>> doc,
> > > >>>>>> but
> > > >>>>>>> still how the token can be obtained (safely) is still open in
> my
> > > >>>> mind.
> > > >>>>> I
> > > >>>>>>> believe the token used by task executors can be created outside
> > of
> > > >>>> it
> > > >>>>> as
> > > >>>>>>> well (I may be wrong here).
> > > >>>>>>>
> > > >>>>>>> #2. Yeah, we would need something similar for triggerers as
> well,
> > > >>>> but
> > > >>>>>> that
> > > >>>>>>> can be done as part of a different AIP
> > > >>>>>>>
> > > >>>>>>> #3. Yeah, I also believe the API should work largely.
> > > >>>>>>>
> > > >>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can work
> > > >>>> with
> > > >>>>>>> dag_bundles and every dag-processor instance would be treated
> as
> > a
> > > >>>> diff
> > > >>>>>>> bundle.
> > > >>>>>>>
> > > >>>>>>> Also, added points around callbacks, as these are also fetched
> > > >>>> directly
> > > >>>>>>> from the DB.
> > > >>>>>>>
> > > >>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <
> jarek@potiuk.com>
> > > >>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>>> A clarification to this - the dag parser today is likely not
> > > >>>>>> protection
> > > >>>>>>>> against a dedicated malicious DAG author, but it does protect
> > > >>>> against
> > > >>>>>>>> casual DB access attempts - the db session is blanked out in
> the
> > > >>>>>> parsing
> > > >>>>>>>> process , as are the env var configs
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > >>>>>>>> -
> > > >>>>>>>> is this perfect no? but it’s much more than no protection
> > > >>>>>>>> Oh absolutely.. This is exactly what we discussed back then in
> > > >>>> March
> > > >>>>> I
> > > >>>>>>>> think - and the way we decided to go for 3.0 with full
> knowledge
> > > >>>> it's
> > > >>>>>> not
> > > >>>>>>>> protecting against all threats.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > > >>>> ash@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> A clarification to this - the dag parser today is likely not
> > > >>>>>> protection
> > > >>>>>>>>> against a dedicated malicious DAG author, but it does protect
> > > >>>>> against
> > > >>>>>>>>> casual DB access attempts - the db session is blanked out in
> > > >>>> the
> > > >>>>>>> parsing
> > > >>>>>>>>> process , as are the env var configs
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > >>>>>>>>> - is this perfect no? but it’s much more than no protection
> > > >>>>>>>>>
> > > >>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
> > > >>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> Currently in the DagFile processor there is no built-in
> > > >>>>> protection
> > > >>>>>>>>> against
> > > >>>>>>>>>> user code from Dag Parsing to - for example - read database
> > > >>>>>>>>>> credentials from airflow configuration and use them to talk
> > > >>>> to DB
> > > >>>>>>>>> directly.
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
>
> My main concern with this right now is the serialisation format of DAGs —
> it wasn’t really designed with remote submission in mind, so it need some
> careful examination to see if it is fit for this purpose or not.
>
I'm not sure on this point, cause if we are able to convert a DAG into
JSON, then it has to be transferable over the internet.
In particular One of the things I worry about is that the JSON can get huge
> — I’ve seem this as large as 10-20Mb for some dags
Yeah, agree on this, thats why we can transfer compressed data instead of
real json. Of course, this won't guarantee that the payload will always be
small enough, but we can't say that it'll definitely happen either.
I also wonder if as part of this proposal we should move the Callback
> requests off the dag parsers and on to the workers instead
let's make such a "workfload" implementation stream that could support both
> - Deadlines and DAG parsing logic
I don't have any strong opinion here, but it feels like it's gonna blow up
the scope of the AIP too much.
On Fri, Aug 1, 2025 at 2:27 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > My main concern with this right now is the serialisation format of DAGs —
> it wasn’t really designed with remote submission in mind, so it need some
> careful examination to see if it is fit for this purpose or not.
>
> Yep. That might be potentially a problem (or at least "need more resources
> to run airflow") and that is where my "2x memory" came from if we do it in
> a trivial way. Currently we a) keep the whole DAG in memory when
> serializing it b) submit it to database (also using essentially some kind
> of API (implemented by the database client) - so we know the whole thing
> "might work" but indeed if you use a trivial implementation of submitting
> the whole json - it basically means that the whole json will have to also
> be kept in the memory of API server. But we also compress it when needed -
> I wonder what are the compression ratios we saw with those 10-20MBs Dags -
> if the problem is using strings where bool would suffice, compression
> should generally help a lot. We could only ever send compressed data over
> the API - there seems to be no need to send "plain JSON" data over the API
> or storing the plain JSON in the DB (of course that trades memory for CPU).
>
> I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have support for
> any kind if binary data streaming - because that could help a lot of if we
> could use streaming HTTP API and chunk and append the binary chunks (when
> writing) - or read data in chunks ans stream them back via the API. That
> could seriously decrease the amount of memory needed by the API server to
> process such huge serialized dags.
>
> And yeah - I would also love the "execute task" to be implemented here -
> but I am not sure if this should be part of the same effort or maybe a
> separate implementation? That sounds very loosely coupled with DB
> isolation. And it seems a common theme - I think that would also make the
> sync Deadline alerts case that we discussed at the dev call today. I wonder
> if that should not be kind of parallel (let's make such a "workfload"
> implementation stream that could support both - Deadlines and DAG parsing
> logic. We have already two "users" for it and I really love the saying "if
> you want to make something reusable - make it usable first" - seems like
> we might have good opportunity to make such workload implementation "doubly
> used" from the beginning which would increase chances it will be
> "reusable" for other things as well :).
>
> J.
>
>
> On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > My main concern with this right now is the serialisation format of DAGs —
> > it wasn’t really designed with remote submission in mind, so it need some
> > careful examination to see if it is fit for this purpose or not.
> >
> > In particular One of the things I worry about is that the JSON can get
> > huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which is
> > likely due to things being included as text when a bool might suffice,
> for
> > example) But I don’t think “just submit the existing JSON over an API”
> is a
> > good idea.
> >
> > I also wonder if as part of this proposal we should move the Callback
> > requests off the dag parsers and on to the workers instead — in AIP-72 we
> > introduced the concept of a Workload, with the only one existing right
> now
> > is “ExecuteTask”
> >
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> > — it might be time to finally move task and dag callbacks to the same
> thing
> > and make dag parsers only responsible for, well, parsing. :)
> >
> > These are all solvable problems, and this will be a great feature to
> have,
> > but we need to do some more thinking and planning first.
> >
> > -ash
> >
> > > On 31 Jul 2025, at 10:12, Sumit Maheshwari <su...@gmail.com>
> > wrote:
> > >
> > > Gentle reminder for everyone to review the proposal.
> > >
> > > Updated link:
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> > >
> > > On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <
> sumeet.manit@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks everyone for reviewing this AIP. As Jarek and others
> suggested, I
> > >> expanded the scope of this AIP and divided it into three phases. With
> > the
> > >> increased scope, the boundary line between this AIP and AIP-85 got a
> > little
> > >> thinner, but I believe these are still two different enhancements to
> > make.
> > >>
> > >>
> > >>
> > >> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> > sumeet.manit@gmail.com>
> > >> wrote:
> > >>
> > >>> Yeah, overall it makes sense to include Triggers as well to be part
> of
> > >>> this AIP and phase out the implementation. Though I didn't exclude
> > Triggers
> > >>> because "Uber" doesn't need that, I just thought of keeping the scope
> > of
> > >>> development small and achieving them, just like it was done in
> Airlfow
> > 3 by
> > >>> secluding only workers and not DAG-processor & Triggers.
> > >>>
> > >>> But if you think Triggers should be part of this AIP itself, then I
> can
> > >>> do that and include Triggers as well in it.
> > >>>
> > >>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>>
> > >>>> I would very much prefer the architectural choices of this AIP are
> > based
> > >>>> on
> > >>>> "general public" needs rather than "Uber needs" even if Uber will be
> > >>>> implementing it - so from my point of view having Trigger separation
> > as
> > >>>> part of it is quite important.
> > >>>>
> > >>>> But that's not even this.
> > >>>>
> > >>>> We've been discussing for example for Deadlines (being implemented
> by
> > >>>> Dennis and Ramit a possibility of short, notification-style
> > "deadlines"
> > >>>> to be send to triggerer for execution - this is well advanced now,
> and
> > >>>> whether you want it or not Dag-provided code might be serialized and
> > sent
> > >>>> to triggerer for execution. This is part of our "broader"
> > architectural
> > >>>> change where we treat "workers" and "triggerer" similarly as a
> general
> > >>>> executors of "sync" and "async" tasks respectively. That's where
> > Airflow
> > >>>> is
> > >>>> evolving towards - inevitably.
> > >>>>
> > >>>> But we can of course phase things in out for implementation - even
> if
> > AIP
> > >>>> should cover both, I think if the goal of the AIP and preamble is
> > about
> > >>>> separating "user code" from "database" as the main reason, it also
> > means
> > >>>> Triggerer if you ask me (from design point of view at least).
> > >>>>
> > >>>> Again implementation can be phased and even different people and
> teams
> > >>>> might work on those phases/pieces.
> > >>>>
> > >>>> J.
> > >>>>
> > >>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> > sumeet.manit@gmail.com
> > >>>>>
> > >>>> wrote:
> > >>>>
> > >>>>>>
> > >>>>>>> #2. Yeah, we would need something similar for triggerers as well,
> > >>>> but
> > >>>>>> that
> > >>>>>> can be done as part of a different AIP
> > >>>>>
> > >>>>>
> > >>>>> You won't achieve your goal of "true" isolation of user code if you
> > >>>> don't
> > >>>>>> do triggerer. I think if the goal is to achieve it - it should
> cover
> > >>>>> both.
> > >>>>>
> > >>>>>
> > >>>>> My bad, should've explained our architecture for triggers as well,
> > >>>>> apologies. So here it is:
> > >>>>>
> > >>>>>
> > >>>>> - Triggers would be running on a centralized service, so all the
> > >>>> Trigger
> > >>>>> classes will be part of the platform team's repo and not the
> > >>>> customer's
> > >>>>> repo
> > >>>>> - The triggers won't be able to use any libs other than std ones,
> > >>>> which
> > >>>>> are being used in core Airflow (like requests, etc)
> > >>>>> - As we are the owners of the core Airflow repo, customers have
> to
> > >>>> get
> > >>>>> our approval to land any class in this path (unlike the dags repo
> > >>>> which
> > >>>>> they own)
> > >>>>> - When a customer's task defer, we would have an allowlist on our
> > >>>> side
> > >>>>> to check if we should do the async polling or not
> > >>>>> - If the Trigger class isn't part of our repo (allowlist), just
> > >>>> fail the
> > >>>>> task, as anyway we won't be having the code that they used in the
> > >>>>> trigger
> > >>>>> class
> > >>>>> - If any of these conditions aren't suitable for you (as a
> > >>>> customer),
> > >>>>> feel free to use sync tasks only
> > >>>>>
> > >>>>>
> > >>>>> But in general, I agree to make triggerer svc also communicate over
> > >>>> apis
> > >>>>> only. If that is done, then we can have instances of triggerer svc
> > >>>> running
> > >>>>> at customer's side as well, which can process any type of trigger
> > >>>> class.
> > >>>>> Though that's not a blocker for us at the moment, cause triggerer
> are
> > >>>>> mostly doing just polling using simple libs like requests.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> > >>>> <ikholopov@google.com.invalid
> > >>>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Thanks Sumit for the detailed proposal. Overall I believe it
> aligns
> > >>>> well
> > >>>>>> with the goals of making Airflow well-scalable beyond a
> single-team
> > >>>>>> deployment (and AIP-85 goals), so you have my full support with
> this
> > >>>> one.
> > >>>>>>
> > >>>>>> I've left a couple of clarification requests on the AIP page.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Igor
> > >>>>>>
> > >>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > >>>>> sumeet.manit@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks Jarek and Ash, for the initial review. It's good to know
> > >>>> that
> > >>>>> the
> > >>>>>>> DAG processor has some preemptive measures in place to prevent
> > >>>> access
> > >>>>>>> to the DB. However, the main issue we are trying to solve is not
> to
> > >>>>>> provide
> > >>>>>>> DB creds to the customer teams, who are using Airflow as a
> > >>>> multi-tenant
> > >>>>>>> orchestration platform. I've updated the doc to reflect this
> point
> > >>>> as
> > >>>>>> well.
> > >>>>>>>
> > >>>>>>> Answering Jarek's points,
> > >>>>>>>
> > >>>>>>> #1. Yeah, had forgot to write about token mechanism, added that
> in
> > >>>> doc,
> > >>>>>> but
> > >>>>>>> still how the token can be obtained (safely) is still open in my
> > >>>> mind.
> > >>>>> I
> > >>>>>>> believe the token used by task executors can be created outside
> of
> > >>>> it
> > >>>>> as
> > >>>>>>> well (I may be wrong here).
> > >>>>>>>
> > >>>>>>> #2. Yeah, we would need something similar for triggerers as well,
> > >>>> but
> > >>>>>> that
> > >>>>>>> can be done as part of a different AIP
> > >>>>>>>
> > >>>>>>> #3. Yeah, I also believe the API should work largely.
> > >>>>>>>
> > >>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can work
> > >>>> with
> > >>>>>>> dag_bundles and every dag-processor instance would be treated as
> a
> > >>>> diff
> > >>>>>>> bundle.
> > >>>>>>>
> > >>>>>>> Also, added points around callbacks, as these are also fetched
> > >>>> directly
> > >>>>>>> from the DB.
> > >>>>>>>
> > >>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
> > >>>>> wrote:
> > >>>>>>>
> > >>>>>>>>> A clarification to this - the dag parser today is likely not
> > >>>>>> protection
> > >>>>>>>> against a dedicated malicious DAG author, but it does protect
> > >>>> against
> > >>>>>>>> casual DB access attempts - the db session is blanked out in the
> > >>>>>> parsing
> > >>>>>>>> process , as are the env var configs
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > >>>>>>>> -
> > >>>>>>>> is this perfect no? but it’s much more than no protection
> > >>>>>>>> Oh absolutely.. This is exactly what we discussed back then in
> > >>>> March
> > >>>>> I
> > >>>>>>>> think - and the way we decided to go for 3.0 with full knowledge
> > >>>> it's
> > >>>>>> not
> > >>>>>>>> protecting against all threats.
> > >>>>>>>>
> > >>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> > >>>> ash@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> A clarification to this - the dag parser today is likely not
> > >>>>>> protection
> > >>>>>>>>> against a dedicated malicious DAG author, but it does protect
> > >>>>> against
> > >>>>>>>>> casual DB access attempts - the db session is blanked out in
> > >>>> the
> > >>>>>>> parsing
> > >>>>>>>>> process , as are the env var configs
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > >>>>>>>>> - is this perfect no? but it’s much more than no protection
> > >>>>>>>>>
> > >>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
> > >>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Currently in the DagFile processor there is no built-in
> > >>>>> protection
> > >>>>>>>>> against
> > >>>>>>>>>> user code from Dag Parsing to - for example - read database
> > >>>>>>>>>> credentials from airflow configuration and use them to talk
> > >>>> to DB
> > >>>>>>>>> directly.
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
> My main concern with this right now is the serialisation format of DAGs —
it wasn’t really designed with remote submission in mind, so it need some
careful examination to see if it is fit for this purpose or not.
Yep. That might be potentially a problem (or at least "need more resources
to run airflow") and that is where my "2x memory" came from if we do it in
a trivial way. Currently we a) keep the whole DAG in memory when
serializing it b) submit it to database (also using essentially some kind
of API (implemented by the database client) - so we know the whole thing
"might work" but indeed if you use a trivial implementation of submitting
the whole json - it basically means that the whole json will have to also
be kept in the memory of API server. But we also compress it when needed -
I wonder what are the compression ratios we saw with those 10-20MBs Dags -
if the problem is using strings where bool would suffice, compression
should generally help a lot. We could only ever send compressed data over
the API - there seems to be no need to send "plain JSON" data over the API
or storing the plain JSON in the DB (of course that trades memory for CPU).
I wonder if sqlalchemy 2 (and drivers for MySQL/Postgres) have support for
any kind if binary data streaming - because that could help a lot of if we
could use streaming HTTP API and chunk and append the binary chunks (when
writing) - or read data in chunks ans stream them back via the API. That
could seriously decrease the amount of memory needed by the API server to
process such huge serialized dags.
And yeah - I would also love the "execute task" to be implemented here -
but I am not sure if this should be part of the same effort or maybe a
separate implementation? That sounds very loosely coupled with DB
isolation. And it seems a common theme - I think that would also make the
sync Deadline alerts case that we discussed at the dev call today. I wonder
if that should not be kind of parallel (let's make such a "workfload"
implementation stream that could support both - Deadlines and DAG parsing
logic. We have already two "users" for it and I really love the saying "if
you want to make something reusable - make it usable first" - seems like
we might have good opportunity to make such workload implementation "doubly
used" from the beginning which would increase chances it will be
"reusable" for other things as well :).
J.
On Thu, Jul 31, 2025 at 12:28 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> My main concern with this right now is the serialisation format of DAGs —
> it wasn’t really designed with remote submission in mind, so it need some
> careful examination to see if it is fit for this purpose or not.
>
> In particular One of the things I worry about is that the JSON can get
> huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which is
> likely due to things being included as text when a bool might suffice, for
> example) But I don’t think “just submit the existing JSON over an API” is a
> good idea.
>
> I also wonder if as part of this proposal we should move the Callback
> requests off the dag parsers and on to the workers instead — in AIP-72 we
> introduced the concept of a Workload, with the only one existing right now
> is “ExecuteTask”
> https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88
> — it might be time to finally move task and dag callbacks to the same thing
> and make dag parsers only responsible for, well, parsing. :)
>
> These are all solvable problems, and this will be a great feature to have,
> but we need to do some more thinking and planning first.
>
> -ash
>
> > On 31 Jul 2025, at 10:12, Sumit Maheshwari <su...@gmail.com>
> wrote:
> >
> > Gentle reminder for everyone to review the proposal.
> >
> > Updated link:
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
> >
> > On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <sumeet.manit@gmail.com
> >
> > wrote:
> >
> >> Thanks everyone for reviewing this AIP. As Jarek and others suggested, I
> >> expanded the scope of this AIP and divided it into three phases. With
> the
> >> increased scope, the boundary line between this AIP and AIP-85 got a
> little
> >> thinner, but I believe these are still two different enhancements to
> make.
> >>
> >>
> >>
> >> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <
> sumeet.manit@gmail.com>
> >> wrote:
> >>
> >>> Yeah, overall it makes sense to include Triggers as well to be part of
> >>> this AIP and phase out the implementation. Though I didn't exclude
> Triggers
> >>> because "Uber" doesn't need that, I just thought of keeping the scope
> of
> >>> development small and achieving them, just like it was done in Airlfow
> 3 by
> >>> secluding only workers and not DAG-processor & Triggers.
> >>>
> >>> But if you think Triggers should be part of this AIP itself, then I can
> >>> do that and include Triggers as well in it.
> >>>
> >>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>> I would very much prefer the architectural choices of this AIP are
> based
> >>>> on
> >>>> "general public" needs rather than "Uber needs" even if Uber will be
> >>>> implementing it - so from my point of view having Trigger separation
> as
> >>>> part of it is quite important.
> >>>>
> >>>> But that's not even this.
> >>>>
> >>>> We've been discussing for example for Deadlines (being implemented by
> >>>> Dennis and Ramit a possibility of short, notification-style
> "deadlines"
> >>>> to be send to triggerer for execution - this is well advanced now, and
> >>>> whether you want it or not Dag-provided code might be serialized and
> sent
> >>>> to triggerer for execution. This is part of our "broader"
> architectural
> >>>> change where we treat "workers" and "triggerer" similarly as a general
> >>>> executors of "sync" and "async" tasks respectively. That's where
> Airflow
> >>>> is
> >>>> evolving towards - inevitably.
> >>>>
> >>>> But we can of course phase things in out for implementation - even if
> AIP
> >>>> should cover both, I think if the goal of the AIP and preamble is
> about
> >>>> separating "user code" from "database" as the main reason, it also
> means
> >>>> Triggerer if you ask me (from design point of view at least).
> >>>>
> >>>> Again implementation can be phased and even different people and teams
> >>>> might work on those phases/pieces.
> >>>>
> >>>> J.
> >>>>
> >>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <
> sumeet.manit@gmail.com
> >>>>>
> >>>> wrote:
> >>>>
> >>>>>>
> >>>>>>> #2. Yeah, we would need something similar for triggerers as well,
> >>>> but
> >>>>>> that
> >>>>>> can be done as part of a different AIP
> >>>>>
> >>>>>
> >>>>> You won't achieve your goal of "true" isolation of user code if you
> >>>> don't
> >>>>>> do triggerer. I think if the goal is to achieve it - it should cover
> >>>>> both.
> >>>>>
> >>>>>
> >>>>> My bad, should've explained our architecture for triggers as well,
> >>>>> apologies. So here it is:
> >>>>>
> >>>>>
> >>>>> - Triggers would be running on a centralized service, so all the
> >>>> Trigger
> >>>>> classes will be part of the platform team's repo and not the
> >>>> customer's
> >>>>> repo
> >>>>> - The triggers won't be able to use any libs other than std ones,
> >>>> which
> >>>>> are being used in core Airflow (like requests, etc)
> >>>>> - As we are the owners of the core Airflow repo, customers have to
> >>>> get
> >>>>> our approval to land any class in this path (unlike the dags repo
> >>>> which
> >>>>> they own)
> >>>>> - When a customer's task defer, we would have an allowlist on our
> >>>> side
> >>>>> to check if we should do the async polling or not
> >>>>> - If the Trigger class isn't part of our repo (allowlist), just
> >>>> fail the
> >>>>> task, as anyway we won't be having the code that they used in the
> >>>>> trigger
> >>>>> class
> >>>>> - If any of these conditions aren't suitable for you (as a
> >>>> customer),
> >>>>> feel free to use sync tasks only
> >>>>>
> >>>>>
> >>>>> But in general, I agree to make triggerer svc also communicate over
> >>>> apis
> >>>>> only. If that is done, then we can have instances of triggerer svc
> >>>> running
> >>>>> at customer's side as well, which can process any type of trigger
> >>>> class.
> >>>>> Though that's not a blocker for us at the moment, cause triggerer are
> >>>>> mostly doing just polling using simple libs like requests.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> >>>> <ikholopov@google.com.invalid
> >>>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks Sumit for the detailed proposal. Overall I believe it aligns
> >>>> well
> >>>>>> with the goals of making Airflow well-scalable beyond a single-team
> >>>>>> deployment (and AIP-85 goals), so you have my full support with this
> >>>> one.
> >>>>>>
> >>>>>> I've left a couple of clarification requests on the AIP page.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Igor
> >>>>>>
> >>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> >>>>> sumeet.manit@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Jarek and Ash, for the initial review. It's good to know
> >>>> that
> >>>>> the
> >>>>>>> DAG processor has some preemptive measures in place to prevent
> >>>> access
> >>>>>>> to the DB. However, the main issue we are trying to solve is not to
> >>>>>> provide
> >>>>>>> DB creds to the customer teams, who are using Airflow as a
> >>>> multi-tenant
> >>>>>>> orchestration platform. I've updated the doc to reflect this point
> >>>> as
> >>>>>> well.
> >>>>>>>
> >>>>>>> Answering Jarek's points,
> >>>>>>>
> >>>>>>> #1. Yeah, had forgot to write about token mechanism, added that in
> >>>> doc,
> >>>>>> but
> >>>>>>> still how the token can be obtained (safely) is still open in my
> >>>> mind.
> >>>>> I
> >>>>>>> believe the token used by task executors can be created outside of
> >>>> it
> >>>>> as
> >>>>>>> well (I may be wrong here).
> >>>>>>>
> >>>>>>> #2. Yeah, we would need something similar for triggerers as well,
> >>>> but
> >>>>>> that
> >>>>>>> can be done as part of a different AIP
> >>>>>>>
> >>>>>>> #3. Yeah, I also believe the API should work largely.
> >>>>>>>
> >>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can work
> >>>> with
> >>>>>>> dag_bundles and every dag-processor instance would be treated as a
> >>>> diff
> >>>>>>> bundle.
> >>>>>>>
> >>>>>>> Also, added points around callbacks, as these are also fetched
> >>>> directly
> >>>>>>> from the DB.
> >>>>>>>
> >>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>>> A clarification to this - the dag parser today is likely not
> >>>>>> protection
> >>>>>>>> against a dedicated malicious DAG author, but it does protect
> >>>> against
> >>>>>>>> casual DB access attempts - the db session is blanked out in the
> >>>>>> parsing
> >>>>>>>> process , as are the env var configs
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >>>>>>>> -
> >>>>>>>> is this perfect no? but it’s much more than no protection
> >>>>>>>> Oh absolutely.. This is exactly what we discussed back then in
> >>>> March
> >>>>> I
> >>>>>>>> think - and the way we decided to go for 3.0 with full knowledge
> >>>> it's
> >>>>>> not
> >>>>>>>> protecting against all threats.
> >>>>>>>>
> >>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
> >>>> ash@apache.org>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> A clarification to this - the dag parser today is likely not
> >>>>>> protection
> >>>>>>>>> against a dedicated malicious DAG author, but it does protect
> >>>>> against
> >>>>>>>>> casual DB access attempts - the db session is blanked out in
> >>>> the
> >>>>>>> parsing
> >>>>>>>>> process , as are the env var configs
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> >>>>>>>>> - is this perfect no? but it’s much more than no protection
> >>>>>>>>>
> >>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
> >>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Currently in the DagFile processor there is no built-in
> >>>>> protection
> >>>>>>>>> against
> >>>>>>>>>> user code from Dag Parsing to - for example - read database
> >>>>>>>>>> credentials from airflow configuration and use them to talk
> >>>> to DB
> >>>>>>>>> directly.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
>
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Ash Berlin-Taylor <as...@apache.org>.
My main concern with this right now is the serialisation format of DAGs — it wasn’t really designed with remote submission in mind, so it need some careful examination to see if it is fit for this purpose or not.
In particular One of the things I worry about is that the JSON can get huge — I’ve seem this as large as 10-20Mb for some dags(!!) (which is likely due to things being included as text when a bool might suffice, for example) But I don’t think “just submit the existing JSON over an API” is a good idea.
I also wonder if as part of this proposal we should move the Callback requests off the dag parsers and on to the workers instead — in AIP-72 we introduced the concept of a Workload, with the only one existing right now is “ExecuteTask” https://github.com/apache/airflow/blob/8e1201c7713d5c677fa6f6d48bbd4f6903505f61/airflow-core/src/airflow/executors/workloads.py#L87-L88 — it might be time to finally move task and dag callbacks to the same thing and make dag parsers only responsible for, well, parsing. :)
These are all solvable problems, and this will be a great feature to have, but we need to do some more thinking and planning first.
-ash
> On 31 Jul 2025, at 10:12, Sumit Maheshwari <su...@gmail.com> wrote:
>
> Gentle reminder for everyone to review the proposal.
>
> Updated link:
> https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
>
> On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
>> Thanks everyone for reviewing this AIP. As Jarek and others suggested, I
>> expanded the scope of this AIP and divided it into three phases. With the
>> increased scope, the boundary line between this AIP and AIP-85 got a little
>> thinner, but I believe these are still two different enhancements to make.
>>
>>
>>
>> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>
>>> Yeah, overall it makes sense to include Triggers as well to be part of
>>> this AIP and phase out the implementation. Though I didn't exclude Triggers
>>> because "Uber" doesn't need that, I just thought of keeping the scope of
>>> development small and achieving them, just like it was done in Airlfow 3 by
>>> secluding only workers and not DAG-processor & Triggers.
>>>
>>> But if you think Triggers should be part of this AIP itself, then I can
>>> do that and include Triggers as well in it.
>>>
>>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> I would very much prefer the architectural choices of this AIP are based
>>>> on
>>>> "general public" needs rather than "Uber needs" even if Uber will be
>>>> implementing it - so from my point of view having Trigger separation as
>>>> part of it is quite important.
>>>>
>>>> But that's not even this.
>>>>
>>>> We've been discussing for example for Deadlines (being implemented by
>>>> Dennis and Ramit a possibility of short, notification-style "deadlines"
>>>> to be send to triggerer for execution - this is well advanced now, and
>>>> whether you want it or not Dag-provided code might be serialized and sent
>>>> to triggerer for execution. This is part of our "broader" architectural
>>>> change where we treat "workers" and "triggerer" similarly as a general
>>>> executors of "sync" and "async" tasks respectively. That's where Airflow
>>>> is
>>>> evolving towards - inevitably.
>>>>
>>>> But we can of course phase things in out for implementation - even if AIP
>>>> should cover both, I think if the goal of the AIP and preamble is about
>>>> separating "user code" from "database" as the main reason, it also means
>>>> Triggerer if you ask me (from design point of view at least).
>>>>
>>>> Again implementation can be phased and even different people and teams
>>>> might work on those phases/pieces.
>>>>
>>>> J.
>>>>
>>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <sumeet.manit@gmail.com
>>>>>
>>>> wrote:
>>>>
>>>>>>
>>>>>>> #2. Yeah, we would need something similar for triggerers as well,
>>>> but
>>>>>> that
>>>>>> can be done as part of a different AIP
>>>>>
>>>>>
>>>>> You won't achieve your goal of "true" isolation of user code if you
>>>> don't
>>>>>> do triggerer. I think if the goal is to achieve it - it should cover
>>>>> both.
>>>>>
>>>>>
>>>>> My bad, should've explained our architecture for triggers as well,
>>>>> apologies. So here it is:
>>>>>
>>>>>
>>>>> - Triggers would be running on a centralized service, so all the
>>>> Trigger
>>>>> classes will be part of the platform team's repo and not the
>>>> customer's
>>>>> repo
>>>>> - The triggers won't be able to use any libs other than std ones,
>>>> which
>>>>> are being used in core Airflow (like requests, etc)
>>>>> - As we are the owners of the core Airflow repo, customers have to
>>>> get
>>>>> our approval to land any class in this path (unlike the dags repo
>>>> which
>>>>> they own)
>>>>> - When a customer's task defer, we would have an allowlist on our
>>>> side
>>>>> to check if we should do the async polling or not
>>>>> - If the Trigger class isn't part of our repo (allowlist), just
>>>> fail the
>>>>> task, as anyway we won't be having the code that they used in the
>>>>> trigger
>>>>> class
>>>>> - If any of these conditions aren't suitable for you (as a
>>>> customer),
>>>>> feel free to use sync tasks only
>>>>>
>>>>>
>>>>> But in general, I agree to make triggerer svc also communicate over
>>>> apis
>>>>> only. If that is done, then we can have instances of triggerer svc
>>>> running
>>>>> at customer's side as well, which can process any type of trigger
>>>> class.
>>>>> Though that's not a blocker for us at the moment, cause triggerer are
>>>>> mostly doing just polling using simple libs like requests.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
>>>> <ikholopov@google.com.invalid
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> Thanks Sumit for the detailed proposal. Overall I believe it aligns
>>>> well
>>>>>> with the goals of making Airflow well-scalable beyond a single-team
>>>>>> deployment (and AIP-85 goals), so you have my full support with this
>>>> one.
>>>>>>
>>>>>> I've left a couple of clarification requests on the AIP page.
>>>>>>
>>>>>> Thanks,
>>>>>> Igor
>>>>>>
>>>>>> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
>>>>> sumeet.manit@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Jarek and Ash, for the initial review. It's good to know
>>>> that
>>>>> the
>>>>>>> DAG processor has some preemptive measures in place to prevent
>>>> access
>>>>>>> to the DB. However, the main issue we are trying to solve is not to
>>>>>> provide
>>>>>>> DB creds to the customer teams, who are using Airflow as a
>>>> multi-tenant
>>>>>>> orchestration platform. I've updated the doc to reflect this point
>>>> as
>>>>>> well.
>>>>>>>
>>>>>>> Answering Jarek's points,
>>>>>>>
>>>>>>> #1. Yeah, had forgot to write about token mechanism, added that in
>>>> doc,
>>>>>> but
>>>>>>> still how the token can be obtained (safely) is still open in my
>>>> mind.
>>>>> I
>>>>>>> believe the token used by task executors can be created outside of
>>>> it
>>>>> as
>>>>>>> well (I may be wrong here).
>>>>>>>
>>>>>>> #2. Yeah, we would need something similar for triggerers as well,
>>>> but
>>>>>> that
>>>>>>> can be done as part of a different AIP
>>>>>>>
>>>>>>> #3. Yeah, I also believe the API should work largely.
>>>>>>>
>>>>>>> #4. Added that in the AIP, that instead of dag_dirs we can work
>>>> with
>>>>>>> dag_bundles and every dag-processor instance would be treated as a
>>>> diff
>>>>>>> bundle.
>>>>>>>
>>>>>>> Also, added points around callbacks, as these are also fetched
>>>> directly
>>>>>>> from the DB.
>>>>>>>
>>>>>>> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
>>>>> wrote:
>>>>>>>
>>>>>>>>> A clarification to this - the dag parser today is likely not
>>>>>> protection
>>>>>>>> against a dedicated malicious DAG author, but it does protect
>>>> against
>>>>>>>> casual DB access attempts - the db session is blanked out in the
>>>>>> parsing
>>>>>>>> process , as are the env var configs
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>>>>>>>> -
>>>>>>>> is this perfect no? but it’s much more than no protection
>>>>>>>> Oh absolutely.. This is exactly what we discussed back then in
>>>> March
>>>>> I
>>>>>>>> think - and the way we decided to go for 3.0 with full knowledge
>>>> it's
>>>>>> not
>>>>>>>> protecting against all threats.
>>>>>>>>
>>>>>>>> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
>>>> ash@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> A clarification to this - the dag parser today is likely not
>>>>>> protection
>>>>>>>>> against a dedicated malicious DAG author, but it does protect
>>>>> against
>>>>>>>>> casual DB access attempts - the db session is blanked out in
>>>> the
>>>>>>> parsing
>>>>>>>>> process , as are the env var configs
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>>>>>>>>> - is this perfect no? but it’s much more than no protection
>>>>>>>>>
>>>>>>>>>> On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Currently in the DagFile processor there is no built-in
>>>>> protection
>>>>>>>>> against
>>>>>>>>>> user code from Dag Parsing to - for example - read database
>>>>>>>>>> credentials from airflow configuration and use them to talk
>>>> to DB
>>>>>>>>> directly.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Gentle reminder for everyone to review the proposal.
Updated link:
https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-92+Isolate+DAG+processor%2C+Callback+processor%2C+and+Triggerer+from+core+services
On Tue, Jul 29, 2025 at 4:37 PM Sumit Maheshwari <su...@gmail.com>
wrote:
> Thanks everyone for reviewing this AIP. As Jarek and others suggested, I
> expanded the scope of this AIP and divided it into three phases. With the
> increased scope, the boundary line between this AIP and AIP-85 got a little
> thinner, but I believe these are still two different enhancements to make.
>
>
>
> On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
>> Yeah, overall it makes sense to include Triggers as well to be part of
>> this AIP and phase out the implementation. Though I didn't exclude Triggers
>> because "Uber" doesn't need that, I just thought of keeping the scope of
>> development small and achieving them, just like it was done in Airlfow 3 by
>> secluding only workers and not DAG-processor & Triggers.
>>
>> But if you think Triggers should be part of this AIP itself, then I can
>> do that and include Triggers as well in it.
>>
>> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> I would very much prefer the architectural choices of this AIP are based
>>> on
>>> "general public" needs rather than "Uber needs" even if Uber will be
>>> implementing it - so from my point of view having Trigger separation as
>>> part of it is quite important.
>>>
>>> But that's not even this.
>>>
>>> We've been discussing for example for Deadlines (being implemented by
>>> Dennis and Ramit a possibility of short, notification-style "deadlines"
>>> to be send to triggerer for execution - this is well advanced now, and
>>> whether you want it or not Dag-provided code might be serialized and sent
>>> to triggerer for execution. This is part of our "broader" architectural
>>> change where we treat "workers" and "triggerer" similarly as a general
>>> executors of "sync" and "async" tasks respectively. That's where Airflow
>>> is
>>> evolving towards - inevitably.
>>>
>>> But we can of course phase things in out for implementation - even if AIP
>>> should cover both, I think if the goal of the AIP and preamble is about
>>> separating "user code" from "database" as the main reason, it also means
>>> Triggerer if you ask me (from design point of view at least).
>>>
>>> Again implementation can be phased and even different people and teams
>>> might work on those phases/pieces.
>>>
>>> J.
>>>
>>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <sumeet.manit@gmail.com
>>> >
>>> wrote:
>>>
>>> > >
>>> > > > #2. Yeah, we would need something similar for triggerers as well,
>>> but
>>> > > that
>>> > > can be done as part of a different AIP
>>> >
>>> >
>>> > You won't achieve your goal of "true" isolation of user code if you
>>> don't
>>> > > do triggerer. I think if the goal is to achieve it - it should cover
>>> > both.
>>> >
>>> >
>>> > My bad, should've explained our architecture for triggers as well,
>>> > apologies. So here it is:
>>> >
>>> >
>>> > - Triggers would be running on a centralized service, so all the
>>> Trigger
>>> > classes will be part of the platform team's repo and not the
>>> customer's
>>> > repo
>>> > - The triggers won't be able to use any libs other than std ones,
>>> which
>>> > are being used in core Airflow (like requests, etc)
>>> > - As we are the owners of the core Airflow repo, customers have to
>>> get
>>> > our approval to land any class in this path (unlike the dags repo
>>> which
>>> > they own)
>>> > - When a customer's task defer, we would have an allowlist on our
>>> side
>>> > to check if we should do the async polling or not
>>> > - If the Trigger class isn't part of our repo (allowlist), just
>>> fail the
>>> > task, as anyway we won't be having the code that they used in the
>>> > trigger
>>> > class
>>> > - If any of these conditions aren't suitable for you (as a
>>> customer),
>>> > feel free to use sync tasks only
>>> >
>>> >
>>> > But in general, I agree to make triggerer svc also communicate over
>>> apis
>>> > only. If that is done, then we can have instances of triggerer svc
>>> running
>>> > at customer's side as well, which can process any type of trigger
>>> class.
>>> > Though that's not a blocker for us at the moment, cause triggerer are
>>> > mostly doing just polling using simple libs like requests.
>>> >
>>> >
>>> >
>>> > On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
>>> <ikholopov@google.com.invalid
>>> > >
>>> > wrote:
>>> >
>>> > > Thanks Sumit for the detailed proposal. Overall I believe it aligns
>>> well
>>> > > with the goals of making Airflow well-scalable beyond a single-team
>>> > > deployment (and AIP-85 goals), so you have my full support with this
>>> one.
>>> > >
>>> > > I've left a couple of clarification requests on the AIP page.
>>> > >
>>> > > Thanks,
>>> > > Igor
>>> > >
>>> > > On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
>>> > sumeet.manit@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Thanks Jarek and Ash, for the initial review. It's good to know
>>> that
>>> > the
>>> > > > DAG processor has some preemptive measures in place to prevent
>>> access
>>> > > > to the DB. However, the main issue we are trying to solve is not to
>>> > > provide
>>> > > > DB creds to the customer teams, who are using Airflow as a
>>> multi-tenant
>>> > > > orchestration platform. I've updated the doc to reflect this point
>>> as
>>> > > well.
>>> > > >
>>> > > > Answering Jarek's points,
>>> > > >
>>> > > > #1. Yeah, had forgot to write about token mechanism, added that in
>>> doc,
>>> > > but
>>> > > > still how the token can be obtained (safely) is still open in my
>>> mind.
>>> > I
>>> > > > believe the token used by task executors can be created outside of
>>> it
>>> > as
>>> > > > well (I may be wrong here).
>>> > > >
>>> > > > #2. Yeah, we would need something similar for triggerers as well,
>>> but
>>> > > that
>>> > > > can be done as part of a different AIP
>>> > > >
>>> > > > #3. Yeah, I also believe the API should work largely.
>>> > > >
>>> > > > #4. Added that in the AIP, that instead of dag_dirs we can work
>>> with
>>> > > > dag_bundles and every dag-processor instance would be treated as a
>>> diff
>>> > > > bundle.
>>> > > >
>>> > > > Also, added points around callbacks, as these are also fetched
>>> directly
>>> > > > from the DB.
>>> > > >
>>> > > > On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
>>> > wrote:
>>> > > >
>>> > > > > > A clarification to this - the dag parser today is likely not
>>> > > protection
>>> > > > > against a dedicated malicious DAG author, but it does protect
>>> against
>>> > > > > casual DB access attempts - the db session is blanked out in the
>>> > > parsing
>>> > > > > process , as are the env var configs
>>> > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>>> > > > > -
>>> > > > > is this perfect no? but it’s much more than no protection
>>> > > > > Oh absolutely.. This is exactly what we discussed back then in
>>> March
>>> > I
>>> > > > > think - and the way we decided to go for 3.0 with full knowledge
>>> it's
>>> > > not
>>> > > > > protecting against all threats.
>>> > > > >
>>> > > > > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <
>>> ash@apache.org>
>>> > > > wrote:
>>> > > > >
>>> > > > > > A clarification to this - the dag parser today is likely not
>>> > > protection
>>> > > > > > against a dedicated malicious DAG author, but it does protect
>>> > against
>>> > > > > > casual DB access attempts - the db session is blanked out in
>>> the
>>> > > > parsing
>>> > > > > > process , as are the env var configs
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>>> > > > > > - is this perfect no? but it’s much more than no protection
>>> > > > > >
>>> > > > > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
>>> wrote:
>>> > > > > > >
>>> > > > > > > Currently in the DagFile processor there is no built-in
>>> > protection
>>> > > > > > against
>>> > > > > > > user code from Dag Parsing to - for example - read database
>>> > > > > > > credentials from airflow configuration and use them to talk
>>> to DB
>>> > > > > > directly.
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Thanks everyone for reviewing this AIP. As Jarek and others suggested, I
expanded the scope of this AIP and divided it into three phases. With the
increased scope, the boundary line between this AIP and AIP-85 got a little
thinner, but I believe these are still two different enhancements to make.
On Fri, Jul 25, 2025 at 10:51 PM Sumit Maheshwari <su...@gmail.com>
wrote:
> Yeah, overall it makes sense to include Triggers as well to be part of
> this AIP and phase out the implementation. Though I didn't exclude Triggers
> because "Uber" doesn't need that, I just thought of keeping the scope of
> development small and achieving them, just like it was done in Airlfow 3 by
> secluding only workers and not DAG-processor & Triggers.
>
> But if you think Triggers should be part of this AIP itself, then I can do
> that and include Triggers as well in it.
>
> On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I would very much prefer the architectural choices of this AIP are based
>> on
>> "general public" needs rather than "Uber needs" even if Uber will be
>> implementing it - so from my point of view having Trigger separation as
>> part of it is quite important.
>>
>> But that's not even this.
>>
>> We've been discussing for example for Deadlines (being implemented by
>> Dennis and Ramit a possibility of short, notification-style "deadlines"
>> to be send to triggerer for execution - this is well advanced now, and
>> whether you want it or not Dag-provided code might be serialized and sent
>> to triggerer for execution. This is part of our "broader" architectural
>> change where we treat "workers" and "triggerer" similarly as a general
>> executors of "sync" and "async" tasks respectively. That's where Airflow
>> is
>> evolving towards - inevitably.
>>
>> But we can of course phase things in out for implementation - even if AIP
>> should cover both, I think if the goal of the AIP and preamble is about
>> separating "user code" from "database" as the main reason, it also means
>> Triggerer if you ask me (from design point of view at least).
>>
>> Again implementation can be phased and even different people and teams
>> might work on those phases/pieces.
>>
>> J.
>>
>> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <su...@gmail.com>
>> wrote:
>>
>> > >
>> > > > #2. Yeah, we would need something similar for triggerers as well,
>> but
>> > > that
>> > > can be done as part of a different AIP
>> >
>> >
>> > You won't achieve your goal of "true" isolation of user code if you
>> don't
>> > > do triggerer. I think if the goal is to achieve it - it should cover
>> > both.
>> >
>> >
>> > My bad, should've explained our architecture for triggers as well,
>> > apologies. So here it is:
>> >
>> >
>> > - Triggers would be running on a centralized service, so all the
>> Trigger
>> > classes will be part of the platform team's repo and not the
>> customer's
>> > repo
>> > - The triggers won't be able to use any libs other than std ones,
>> which
>> > are being used in core Airflow (like requests, etc)
>> > - As we are the owners of the core Airflow repo, customers have to
>> get
>> > our approval to land any class in this path (unlike the dags repo
>> which
>> > they own)
>> > - When a customer's task defer, we would have an allowlist on our
>> side
>> > to check if we should do the async polling or not
>> > - If the Trigger class isn't part of our repo (allowlist), just fail
>> the
>> > task, as anyway we won't be having the code that they used in the
>> > trigger
>> > class
>> > - If any of these conditions aren't suitable for you (as a customer),
>> > feel free to use sync tasks only
>> >
>> >
>> > But in general, I agree to make triggerer svc also communicate over apis
>> > only. If that is done, then we can have instances of triggerer svc
>> running
>> > at customer's side as well, which can process any type of trigger class.
>> > Though that's not a blocker for us at the moment, cause triggerer are
>> > mostly doing just polling using simple libs like requests.
>> >
>> >
>> >
>> > On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
>> <ikholopov@google.com.invalid
>> > >
>> > wrote:
>> >
>> > > Thanks Sumit for the detailed proposal. Overall I believe it aligns
>> well
>> > > with the goals of making Airflow well-scalable beyond a single-team
>> > > deployment (and AIP-85 goals), so you have my full support with this
>> one.
>> > >
>> > > I've left a couple of clarification requests on the AIP page.
>> > >
>> > > Thanks,
>> > > Igor
>> > >
>> > > On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
>> > sumeet.manit@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks Jarek and Ash, for the initial review. It's good to know that
>> > the
>> > > > DAG processor has some preemptive measures in place to prevent
>> access
>> > > > to the DB. However, the main issue we are trying to solve is not to
>> > > provide
>> > > > DB creds to the customer teams, who are using Airflow as a
>> multi-tenant
>> > > > orchestration platform. I've updated the doc to reflect this point
>> as
>> > > well.
>> > > >
>> > > > Answering Jarek's points,
>> > > >
>> > > > #1. Yeah, had forgot to write about token mechanism, added that in
>> doc,
>> > > but
>> > > > still how the token can be obtained (safely) is still open in my
>> mind.
>> > I
>> > > > believe the token used by task executors can be created outside of
>> it
>> > as
>> > > > well (I may be wrong here).
>> > > >
>> > > > #2. Yeah, we would need something similar for triggerers as well,
>> but
>> > > that
>> > > > can be done as part of a different AIP
>> > > >
>> > > > #3. Yeah, I also believe the API should work largely.
>> > > >
>> > > > #4. Added that in the AIP, that instead of dag_dirs we can work with
>> > > > dag_bundles and every dag-processor instance would be treated as a
>> diff
>> > > > bundle.
>> > > >
>> > > > Also, added points around callbacks, as these are also fetched
>> directly
>> > > > from the DB.
>> > > >
>> > > > On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
>> > wrote:
>> > > >
>> > > > > > A clarification to this - the dag parser today is likely not
>> > > protection
>> > > > > against a dedicated malicious DAG author, but it does protect
>> against
>> > > > > casual DB access attempts - the db session is blanked out in the
>> > > parsing
>> > > > > process , as are the env var configs
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>> > > > > -
>> > > > > is this perfect no? but it’s much more than no protection
>> > > > > Oh absolutely.. This is exactly what we discussed back then in
>> March
>> > I
>> > > > > think - and the way we decided to go for 3.0 with full knowledge
>> it's
>> > > not
>> > > > > protecting against all threats.
>> > > > >
>> > > > > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <ash@apache.org
>> >
>> > > > wrote:
>> > > > >
>> > > > > > A clarification to this - the dag parser today is likely not
>> > > protection
>> > > > > > against a dedicated malicious DAG author, but it does protect
>> > against
>> > > > > > casual DB access attempts - the db session is blanked out in the
>> > > > parsing
>> > > > > > process , as are the env var configs
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
>> > > > > > - is this perfect no? but it’s much more than no protection
>> > > > > >
>> > > > > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> > > > > > >
>> > > > > > > Currently in the DagFile processor there is no built-in
>> > protection
>> > > > > > against
>> > > > > > > user code from Dag Parsing to - for example - read database
>> > > > > > > credentials from airflow configuration and use them to talk
>> to DB
>> > > > > > directly.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Yeah, overall it makes sense to include Triggers as well to be part of this
AIP and phase out the implementation. Though I didn't exclude Triggers
because "Uber" doesn't need that, I just thought of keeping the scope of
development small and achieving them, just like it was done in Airlfow 3 by
secluding only workers and not DAG-processor & Triggers.
But if you think Triggers should be part of this AIP itself, then I can do
that and include Triggers as well in it.
On Fri, Jul 25, 2025 at 7:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I would very much prefer the architectural choices of this AIP are based on
> "general public" needs rather than "Uber needs" even if Uber will be
> implementing it - so from my point of view having Trigger separation as
> part of it is quite important.
>
> But that's not even this.
>
> We've been discussing for example for Deadlines (being implemented by
> Dennis and Ramit a possibility of short, notification-style "deadlines"
> to be send to triggerer for execution - this is well advanced now, and
> whether you want it or not Dag-provided code might be serialized and sent
> to triggerer for execution. This is part of our "broader" architectural
> change where we treat "workers" and "triggerer" similarly as a general
> executors of "sync" and "async" tasks respectively. That's where Airflow is
> evolving towards - inevitably.
>
> But we can of course phase things in out for implementation - even if AIP
> should cover both, I think if the goal of the AIP and preamble is about
> separating "user code" from "database" as the main reason, it also means
> Triggerer if you ask me (from design point of view at least).
>
> Again implementation can be phased and even different people and teams
> might work on those phases/pieces.
>
> J.
>
> On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
> > >
> > > > #2. Yeah, we would need something similar for triggerers as well, but
> > > that
> > > can be done as part of a different AIP
> >
> >
> > You won't achieve your goal of "true" isolation of user code if you don't
> > > do triggerer. I think if the goal is to achieve it - it should cover
> > both.
> >
> >
> > My bad, should've explained our architecture for triggers as well,
> > apologies. So here it is:
> >
> >
> > - Triggers would be running on a centralized service, so all the
> Trigger
> > classes will be part of the platform team's repo and not the
> customer's
> > repo
> > - The triggers won't be able to use any libs other than std ones,
> which
> > are being used in core Airflow (like requests, etc)
> > - As we are the owners of the core Airflow repo, customers have to get
> > our approval to land any class in this path (unlike the dags repo
> which
> > they own)
> > - When a customer's task defer, we would have an allowlist on our side
> > to check if we should do the async polling or not
> > - If the Trigger class isn't part of our repo (allowlist), just fail
> the
> > task, as anyway we won't be having the code that they used in the
> > trigger
> > class
> > - If any of these conditions aren't suitable for you (as a customer),
> > feel free to use sync tasks only
> >
> >
> > But in general, I agree to make triggerer svc also communicate over apis
> > only. If that is done, then we can have instances of triggerer svc
> running
> > at customer's side as well, which can process any type of trigger class.
> > Though that's not a blocker for us at the moment, cause triggerer are
> > mostly doing just polling using simple libs like requests.
> >
> >
> >
> > On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov
> <ikholopov@google.com.invalid
> > >
> > wrote:
> >
> > > Thanks Sumit for the detailed proposal. Overall I believe it aligns
> well
> > > with the goals of making Airflow well-scalable beyond a single-team
> > > deployment (and AIP-85 goals), so you have my full support with this
> one.
> > >
> > > I've left a couple of clarification requests on the AIP page.
> > >
> > > Thanks,
> > > Igor
> > >
> > > On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> > sumeet.manit@gmail.com>
> > > wrote:
> > >
> > > > Thanks Jarek and Ash, for the initial review. It's good to know that
> > the
> > > > DAG processor has some preemptive measures in place to prevent access
> > > > to the DB. However, the main issue we are trying to solve is not to
> > > provide
> > > > DB creds to the customer teams, who are using Airflow as a
> multi-tenant
> > > > orchestration platform. I've updated the doc to reflect this point as
> > > well.
> > > >
> > > > Answering Jarek's points,
> > > >
> > > > #1. Yeah, had forgot to write about token mechanism, added that in
> doc,
> > > but
> > > > still how the token can be obtained (safely) is still open in my
> mind.
> > I
> > > > believe the token used by task executors can be created outside of it
> > as
> > > > well (I may be wrong here).
> > > >
> > > > #2. Yeah, we would need something similar for triggerers as well, but
> > > that
> > > > can be done as part of a different AIP
> > > >
> > > > #3. Yeah, I also believe the API should work largely.
> > > >
> > > > #4. Added that in the AIP, that instead of dag_dirs we can work with
> > > > dag_bundles and every dag-processor instance would be treated as a
> diff
> > > > bundle.
> > > >
> > > > Also, added points around callbacks, as these are also fetched
> directly
> > > > from the DB.
> > > >
> > > > On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >
> > > > > > A clarification to this - the dag parser today is likely not
> > > protection
> > > > > against a dedicated malicious DAG author, but it does protect
> against
> > > > > casual DB access attempts - the db session is blanked out in the
> > > parsing
> > > > > process , as are the env var configs
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > > -
> > > > > is this perfect no? but it’s much more than no protection
> > > > > Oh absolutely.. This is exactly what we discussed back then in
> March
> > I
> > > > > think - and the way we decided to go for 3.0 with full knowledge
> it's
> > > not
> > > > > protecting against all threats.
> > > > >
> > > > > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org>
> > > > wrote:
> > > > >
> > > > > > A clarification to this - the dag parser today is likely not
> > > protection
> > > > > > against a dedicated malicious DAG author, but it does protect
> > against
> > > > > > casual DB access attempts - the db session is blanked out in the
> > > > parsing
> > > > > > process , as are the env var configs
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > > > - is this perfect no? but it’s much more than no protection
> > > > > >
> > > > > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > > > >
> > > > > > > Currently in the DagFile processor there is no built-in
> > protection
> > > > > > against
> > > > > > > user code from Dag Parsing to - for example - read database
> > > > > > > credentials from airflow configuration and use them to talk to
> DB
> > > > > > directly.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
I would very much prefer the architectural choices of this AIP are based on
"general public" needs rather than "Uber needs" even if Uber will be
implementing it - so from my point of view having Trigger separation as
part of it is quite important.
But that's not even this.
We've been discussing for example for Deadlines (being implemented by
Dennis and Ramit a possibility of short, notification-style "deadlines"
to be send to triggerer for execution - this is well advanced now, and
whether you want it or not Dag-provided code might be serialized and sent
to triggerer for execution. This is part of our "broader" architectural
change where we treat "workers" and "triggerer" similarly as a general
executors of "sync" and "async" tasks respectively. That's where Airflow is
evolving towards - inevitably.
But we can of course phase things in out for implementation - even if AIP
should cover both, I think if the goal of the AIP and preamble is about
separating "user code" from "database" as the main reason, it also means
Triggerer if you ask me (from design point of view at least).
Again implementation can be phased and even different people and teams
might work on those phases/pieces.
J.
On Fri, Jul 25, 2025 at 2:29 PM Sumit Maheshwari <su...@gmail.com>
wrote:
> >
> > > #2. Yeah, we would need something similar for triggerers as well, but
> > that
> > can be done as part of a different AIP
>
>
> You won't achieve your goal of "true" isolation of user code if you don't
> > do triggerer. I think if the goal is to achieve it - it should cover
> both.
>
>
> My bad, should've explained our architecture for triggers as well,
> apologies. So here it is:
>
>
> - Triggers would be running on a centralized service, so all the Trigger
> classes will be part of the platform team's repo and not the customer's
> repo
> - The triggers won't be able to use any libs other than std ones, which
> are being used in core Airflow (like requests, etc)
> - As we are the owners of the core Airflow repo, customers have to get
> our approval to land any class in this path (unlike the dags repo which
> they own)
> - When a customer's task defer, we would have an allowlist on our side
> to check if we should do the async polling or not
> - If the Trigger class isn't part of our repo (allowlist), just fail the
> task, as anyway we won't be having the code that they used in the
> trigger
> class
> - If any of these conditions aren't suitable for you (as a customer),
> feel free to use sync tasks only
>
>
> But in general, I agree to make triggerer svc also communicate over apis
> only. If that is done, then we can have instances of triggerer svc running
> at customer's side as well, which can process any type of trigger class.
> Though that's not a blocker for us at the moment, cause triggerer are
> mostly doing just polling using simple libs like requests.
>
>
>
> On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov <ikholopov@google.com.invalid
> >
> wrote:
>
> > Thanks Sumit for the detailed proposal. Overall I believe it aligns well
> > with the goals of making Airflow well-scalable beyond a single-team
> > deployment (and AIP-85 goals), so you have my full support with this one.
> >
> > I've left a couple of clarification requests on the AIP page.
> >
> > Thanks,
> > Igor
> >
> > On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <
> sumeet.manit@gmail.com>
> > wrote:
> >
> > > Thanks Jarek and Ash, for the initial review. It's good to know that
> the
> > > DAG processor has some preemptive measures in place to prevent access
> > > to the DB. However, the main issue we are trying to solve is not to
> > provide
> > > DB creds to the customer teams, who are using Airflow as a multi-tenant
> > > orchestration platform. I've updated the doc to reflect this point as
> > well.
> > >
> > > Answering Jarek's points,
> > >
> > > #1. Yeah, had forgot to write about token mechanism, added that in doc,
> > but
> > > still how the token can be obtained (safely) is still open in my mind.
> I
> > > believe the token used by task executors can be created outside of it
> as
> > > well (I may be wrong here).
> > >
> > > #2. Yeah, we would need something similar for triggerers as well, but
> > that
> > > can be done as part of a different AIP
> > >
> > > #3. Yeah, I also believe the API should work largely.
> > >
> > > #4. Added that in the AIP, that instead of dag_dirs we can work with
> > > dag_bundles and every dag-processor instance would be treated as a diff
> > > bundle.
> > >
> > > Also, added points around callbacks, as these are also fetched directly
> > > from the DB.
> > >
> > > On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >
> > > > > A clarification to this - the dag parser today is likely not
> > protection
> > > > against a dedicated malicious DAG author, but it does protect against
> > > > casual DB access attempts - the db session is blanked out in the
> > parsing
> > > > process , as are the env var configs
> > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > -
> > > > is this perfect no? but it’s much more than no protection
> > > > Oh absolutely.. This is exactly what we discussed back then in March
> I
> > > > think - and the way we decided to go for 3.0 with full knowledge it's
> > not
> > > > protecting against all threats.
> > > >
> > > > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > >
> > > > > A clarification to this - the dag parser today is likely not
> > protection
> > > > > against a dedicated malicious DAG author, but it does protect
> against
> > > > > casual DB access attempts - the db session is blanked out in the
> > > parsing
> > > > > process , as are the env var configs
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > > - is this perfect no? but it’s much more than no protection
> > > > >
> > > > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > > > >
> > > > > > Currently in the DagFile processor there is no built-in
> protection
> > > > > against
> > > > > > user code from Dag Parsing to - for example - read database
> > > > > > credentials from airflow configuration and use them to talk to DB
> > > > > directly.
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
>
> > #2. Yeah, we would need something similar for triggerers as well, but
> that
> can be done as part of a different AIP
You won't achieve your goal of "true" isolation of user code if you don't
> do triggerer. I think if the goal is to achieve it - it should cover both.
My bad, should've explained our architecture for triggers as well,
apologies. So here it is:
- Triggers would be running on a centralized service, so all the Trigger
classes will be part of the platform team's repo and not the customer's repo
- The triggers won't be able to use any libs other than std ones, which
are being used in core Airflow (like requests, etc)
- As we are the owners of the core Airflow repo, customers have to get
our approval to land any class in this path (unlike the dags repo which
they own)
- When a customer's task defer, we would have an allowlist on our side
to check if we should do the async polling or not
- If the Trigger class isn't part of our repo (allowlist), just fail the
task, as anyway we won't be having the code that they used in the trigger
class
- If any of these conditions aren't suitable for you (as a customer),
feel free to use sync tasks only
But in general, I agree to make triggerer svc also communicate over apis
only. If that is done, then we can have instances of triggerer svc running
at customer's side as well, which can process any type of trigger class.
Though that's not a blocker for us at the moment, cause triggerer are
mostly doing just polling using simple libs like requests.
On Fri, Jul 25, 2025 at 5:03 PM Igor Kholopov <ik...@google.com.invalid>
wrote:
> Thanks Sumit for the detailed proposal. Overall I believe it aligns well
> with the goals of making Airflow well-scalable beyond a single-team
> deployment (and AIP-85 goals), so you have my full support with this one.
>
> I've left a couple of clarification requests on the AIP page.
>
> Thanks,
> Igor
>
> On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <su...@gmail.com>
> wrote:
>
> > Thanks Jarek and Ash, for the initial review. It's good to know that the
> > DAG processor has some preemptive measures in place to prevent access
> > to the DB. However, the main issue we are trying to solve is not to
> provide
> > DB creds to the customer teams, who are using Airflow as a multi-tenant
> > orchestration platform. I've updated the doc to reflect this point as
> well.
> >
> > Answering Jarek's points,
> >
> > #1. Yeah, had forgot to write about token mechanism, added that in doc,
> but
> > still how the token can be obtained (safely) is still open in my mind. I
> > believe the token used by task executors can be created outside of it as
> > well (I may be wrong here).
> >
> > #2. Yeah, we would need something similar for triggerers as well, but
> that
> > can be done as part of a different AIP
> >
> > #3. Yeah, I also believe the API should work largely.
> >
> > #4. Added that in the AIP, that instead of dag_dirs we can work with
> > dag_bundles and every dag-processor instance would be treated as a diff
> > bundle.
> >
> > Also, added points around callbacks, as these are also fetched directly
> > from the DB.
> >
> > On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > A clarification to this - the dag parser today is likely not
> protection
> > > against a dedicated malicious DAG author, but it does protect against
> > > casual DB access attempts - the db session is blanked out in the
> parsing
> > > process , as are the env var configs
> > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > -
> > > is this perfect no? but it’s much more than no protection
> > > Oh absolutely.. This is exactly what we discussed back then in March I
> > > think - and the way we decided to go for 3.0 with full knowledge it's
> not
> > > protecting against all threats.
> > >
> > > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > > > A clarification to this - the dag parser today is likely not
> protection
> > > > against a dedicated malicious DAG author, but it does protect against
> > > > casual DB access attempts - the db session is blanked out in the
> > parsing
> > > > process , as are the env var configs
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > > - is this perfect no? but it’s much more than no protection
> > > >
> > > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > > >
> > > > > Currently in the DagFile processor there is no built-in protection
> > > > against
> > > > > user code from Dag Parsing to - for example - read database
> > > > > credentials from airflow configuration and use them to talk to DB
> > > > directly.
> > > >
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Igor Kholopov <ik...@google.com.INVALID>.
Thanks Sumit for the detailed proposal. Overall I believe it aligns well
with the goals of making Airflow well-scalable beyond a single-team
deployment (and AIP-85 goals), so you have my full support with this one.
I've left a couple of clarification requests on the AIP page.
Thanks,
Igor
On Fri, Jul 25, 2025 at 11:50 AM Sumit Maheshwari <su...@gmail.com>
wrote:
> Thanks Jarek and Ash, for the initial review. It's good to know that the
> DAG processor has some preemptive measures in place to prevent access
> to the DB. However, the main issue we are trying to solve is not to provide
> DB creds to the customer teams, who are using Airflow as a multi-tenant
> orchestration platform. I've updated the doc to reflect this point as well.
>
> Answering Jarek's points,
>
> #1. Yeah, had forgot to write about token mechanism, added that in doc, but
> still how the token can be obtained (safely) is still open in my mind. I
> believe the token used by task executors can be created outside of it as
> well (I may be wrong here).
>
> #2. Yeah, we would need something similar for triggerers as well, but that
> can be done as part of a different AIP
>
> #3. Yeah, I also believe the API should work largely.
>
> #4. Added that in the AIP, that instead of dag_dirs we can work with
> dag_bundles and every dag-processor instance would be treated as a diff
> bundle.
>
> Also, added points around callbacks, as these are also fetched directly
> from the DB.
>
> On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > A clarification to this - the dag parser today is likely not protection
> > against a dedicated malicious DAG author, but it does protect against
> > casual DB access attempts - the db session is blanked out in the parsing
> > process , as are the env var configs
> >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > -
> > is this perfect no? but it’s much more than no protection
> > Oh absolutely.. This is exactly what we discussed back then in March I
> > think - and the way we decided to go for 3.0 with full knowledge it's not
> > protecting against all threats.
> >
> > On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > A clarification to this - the dag parser today is likely not protection
> > > against a dedicated malicious DAG author, but it does protect against
> > > casual DB access attempts - the db session is blanked out in the
> parsing
> > > process , as are the env var configs
> > >
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > > - is this perfect no? but it’s much more than no protection
> > >
> > > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > Currently in the DagFile processor there is no built-in protection
> > > against
> > > > user code from Dag Parsing to - for example - read database
> > > > credentials from airflow configuration and use them to talk to DB
> > > directly.
> > >
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Sumit Maheshwari <su...@gmail.com>.
Thanks Jarek and Ash, for the initial review. It's good to know that the
DAG processor has some preemptive measures in place to prevent access
to the DB. However, the main issue we are trying to solve is not to provide
DB creds to the customer teams, who are using Airflow as a multi-tenant
orchestration platform. I've updated the doc to reflect this point as well.
Answering Jarek's points,
#1. Yeah, had forgot to write about token mechanism, added that in doc, but
still how the token can be obtained (safely) is still open in my mind. I
believe the token used by task executors can be created outside of it as
well (I may be wrong here).
#2. Yeah, we would need something similar for triggerers as well, but that
can be done as part of a different AIP
#3. Yeah, I also believe the API should work largely.
#4. Added that in the AIP, that instead of dag_dirs we can work with
dag_bundles and every dag-processor instance would be treated as a diff
bundle.
Also, added points around callbacks, as these are also fetched directly
from the DB.
On Fri, Jul 25, 2025 at 11:58 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > A clarification to this - the dag parser today is likely not protection
> against a dedicated malicious DAG author, but it does protect against
> casual DB access attempts - the db session is blanked out in the parsing
> process , as are the env var configs
>
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> -
> is this perfect no? but it’s much more than no protection
> Oh absolutely.. This is exactly what we discussed back then in March I
> think - and the way we decided to go for 3.0 with full knowledge it's not
> protecting against all threats.
>
> On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > A clarification to this - the dag parser today is likely not protection
> > against a dedicated malicious DAG author, but it does protect against
> > casual DB access attempts - the db session is blanked out in the parsing
> > process , as are the env var configs
> >
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> > - is this perfect no? but it’s much more than no protection
> >
> > > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > Currently in the DagFile processor there is no built-in protection
> > against
> > > user code from Dag Parsing to - for example - read database
> > > credentials from airflow configuration and use them to talk to DB
> > directly.
> >
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
> A clarification to this - the dag parser today is likely not protection
against a dedicated malicious DAG author, but it does protect against
casual DB access attempts - the db session is blanked out in the parsing
process , as are the env var configs
https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
-
is this perfect no? but it’s much more than no protection
Oh absolutely.. This is exactly what we discussed back then in March I
think - and the way we decided to go for 3.0 with full knowledge it's not
protecting against all threats.
On Fri, Jul 25, 2025 at 8:22 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> A clarification to this - the dag parser today is likely not protection
> against a dedicated malicious DAG author, but it does protect against
> casual DB access attempts - the db session is blanked out in the parsing
> process , as are the env var configs
> https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316
> - is this perfect no? but it’s much more than no protection
>
> > On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > Currently in the DagFile processor there is no built-in protection
> against
> > user code from Dag Parsing to - for example - read database
> > credentials from airflow configuration and use them to talk to DB
> directly.
>
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Ash Berlin-Taylor <as...@apache.org>.
A clarification to this - the dag parser today is likely not protection against a dedicated malicious DAG author, but it does protect against casual DB access attempts - the db session is blanked out in the parsing process , as are the env var configs https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/execution_time/supervisor.py#L274-L316 - is this perfect no? but it’s much more than no protection
> On 24 Jul 2025, at 21:56, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> Currently in the DagFile processor there is no built-in protection against
> user code from Dag Parsing to - for example - read database
> credentials from airflow configuration and use them to talk to DB directly.
Re: [DISCUSS] AIP-92 Isolate DAG parsing logic
Posted by Jarek Potiuk <ja...@potiuk.com>.
In general - yes. That's something we discussed as a next step and "final"
step of true separation of "user code" from "airflow code".
Currently in the DagFile processor there is no built-in protection against
user code from Dag Parsing to - for example - read database
credentials from airflow configuration and use them to talk to DB directly.
I raised the same questions at the dev calls some time ago when we
discussed task isolation - and from the discussion on one of the dev call
this was what we agreed we should have eventually and that was a conscious
choice to (for now) make sure that DB is not actually used in the parsing
process - and parsed serialized dags are passed still through
multiprocessing to a process that has DB access. The deliberate choice was
that 3.0 was only isolating "workers" and nothing else.
This will also make multi-team setup more isolated as well. And especially
if you would like to take it on, that would be cool :)
I would love to first hear feedback on the general concept without diving
in detail comments but my first few - very crude - comments are:
1) there should be a way to authenticate the Dag processor to the API
server (long living auth information). Currently Dag Processor uses DB
credentials to write to the DB (which is not isolated by definition) - but
when we open it as an API we should not allow "anyone" to post new
serialized dags. so there should be a mechanism where we authenticate "dag
processor". Don't have concrete proposal yet - but we need to figure it out
2) I am afraid - Dag processor is not enough. we also need to make a very
similar split in Triggerer. Triggerer is another component that has access
to both - user-code and database. This means "/triggerer" namespace as
well. This api should be used to update/retrieve serialized triggers in
Airflow DB.
3) I don't think performance is a "huge" issue. For POST - Currently DAG
processor performs UPDATE on serialized dag table with the same payload.
The worst case is that we need more api-servers to handle it because
instead of dag -> DB, the payload will be sent via api_server. It could
even be a dedicated api_server if needs be. Yep. It will likely require a
bit more resources - but I think it's just max 2x dag processor memory in
the worst case, which is a "fixed" increase (and likely we could leave the
option of **not** using API server in simple installation - more resources
will be needed only when "more isolation" is needed. I don't think we will
have problem with pagination of GET - there is absolutely no need to
retrieve full serialized dag in the GET method - we only need meta-data,
version information, hash of the serialized dag - but not the serialized
dag itself. That should be rather small especially if point 4).
4) the API should have "dag_bundle" or a set of dag_bundles as primary
"selection" criteria. That would also play very well with the multi-team
where you should have a separate dag processor for a group of dag_bundles
belonging to one team. This also means that the authentication in point 1)
should be covering access to specific dag bundles only - for example I
imagine that we could generate long-living private credentials containing
a signed list of dag bundles that the dag processor should be allowed to
interact with.
J.
On Thu, Jul 24, 2025 at 7:30 PM Sumit Maheshwari <su...@gmail.com>
wrote:
> Hello everyone,
>
> I have created a draft version to separate the running of DAG processor
> from Airflow's core services and moved it closer to the execution side.
>
> Please review the proposal and provide your valuable feedback.
>
> PS: We are in the process of adopting Airflow3 from some in-house setup, so
> I might not have a full understanding of the latest Airflow concepts and
> some other nitty-gritties.
>
> Thanks,
> Sumit
>