You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by "Beck, Vincent" <vi...@amazon.com.INVALID> on 2024/07/25 20:55:33 UTC
[DISCUSS] External event driven scheduling in Airflow
Hello everyone,
I created a draft AIP regarding "External event driven scheduling in Airflow". This proposal is about adding capability in Airflow to schedule DAGs based on external events. Here are some examples of such external events:
- A user signs up to one of the user pool defined in my cloud provider
- One of the databases used in my company has been updated
- A job in my cloud provider has been executed successfully
The intent of this AIP is to leverage datasets (which will be soon assets) and update them based on external events. I would like to propose this AIP for discussion and more importantly, hear some feedbacks from you :)
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
Vincent
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Thanks for the update Vincent!
I also referenced a comment earlier about leveraging patterns for the 80-20
use cases in the poll-model.
However, after seeing how you have defined the syntax for "Asset watchers",
I believe that's already covered in your proposal.
So, all good from my perspective.
Vikram
On Fri, Aug 2, 2024 at 12:29 PM Vincent Beck <vi...@apache.org> wrote:
> Hello everyone,
>
> After multiple conversations with different actors, I decided to move the
> push based event scheduling out of scope of this AIP. I could not find a
> solution for the push based approach that was satisfactory and I think more
> investigation and data from Airflow users is needed to move on with this
> approach. In order to not block the poll based event scheduling, which did
> not receive any concern, I decided to focus AIP-82 on the poll-based event
> scheduling. I updated the AIP accordingly.
>
> Since the poll-based approach did not receive any concerns (and many
> positive feedbacks), I'll start voting in another thread.
>
> Thanks for your feedbacks!
>
> Vincent
>
> On 2024/08/02 00:50:35 Pavankumar Gopidesu wrote:
> > Thanks Vincent and Kaxil, Agree it can be added to future work.
> >
> > Looking at discussion on event mapping, Today we know the source systems
> > and we are aware of the schemas from source systems (S3, SQS, Event Arc
> > etc;)
> >
> > However, as new services emerge and their schemas evolve, this may
> change.
> >
> > Agree with Kaxil has mentioned that transformers are very effective in
> > converting the required format before sending data to target systems.
> >
> > Is it a good idea to define a schema for events to enforce from airflow?
> > There are some open standards, such as CloudEvents[1], that we could
> > consider.
> >
> > This approach would allow us to maintain versions (v1, v2, ... vn) as we
> > evolve, providing utilities that users can easily plug into and send
> events
> > to Airflow.
> >
> > It would also simplify maintenance within the Airflow system to handle
> > incoming events because we get a known version of the schema from
> > user request.
> >
> >
> > If this is already being addressed, please disregard my comments.
> >
> >
> > [1]: https://github.com/cloudevents/spec
> >
> >
> > Regards,
> >
> > Pavan
> >
> > On Fri, Aug 2, 2024 at 1:40 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Yes but the big difference is you will create a single user for
> EventBridge
> > > since that is the one sending request to Airflow, single user for
> EventArc
> > > for GCP and 1 user for every other EventListener or application ---- as
> > > compared to 1 user per type of Payload (since Airflow will need to
> > > understand the payload of the original source). So in that case, you
> will
> > > have 1 user,function mapping for S3, 1user,function mapping for
> Redshift,
> > > and it goes on. The former approach is also consistent with our
> Connection
> > > model, where we have one standardized AWS & GCP connection that works
> for
> > > most, if not all, services.
> > >
> > > but the user will have to be added anyway (some kind of
> > > > service account - because the API needs to be authorized - that part
> is
> > > not
> > > > changed). (unless of course you want to use the same user for all
> kinds
> > > of
> > > > external interfaces, which for security point of view is a very bad
> idea
> > > -
> > > > each external system should have their own "service account" - that
> 's
> > > the
> > > > best practices from the security point of view.
> > >
> > >
> > > On Fri, 2 Aug 2024 at 01:22, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > I am all for it - if we want to stick to event bridge or similar and
> > > > recommend it to our users, it's perfectly fine for me, It would be
> great
> > > > however to add a documentation explaining the steps and some
> examples -
> > > > ideally for most of our providers and "standard" ways of triggering
> such
> > > an
> > > > event. This is even what I proposed originally whent the first
> version of
> > > > the document was created ( to just document how to map the events
> > > > externally).
> > > >
> > > > BTW. Yes - in this case you need to implement the logic in the
> > > > event bridge, but the user will have to be added anyway (some kind of
> > > > service account - because the API needs to be authorized - that part
> is
> > > not
> > > > changed). (unless of course you want to use the same user for all
> kinds
> > > of
> > > > external interfaces, which for security point of view is a very bad
> idea
> > > -
> > > > each external system should have their own "service account" - that
> 's
> > > the
> > > > best practices from the security point of view.
> > > >
> > > > J,
> > > >
> > > > On Fri, Aug 2, 2024 at 2:01 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > > >
> > > > > I was discussing this with Vincent. In either case, same as now or
> the
> > > > one
> > > > > proposed in the AIP, a user will have to use something like AWS
> > > > > EventBridge[1] or GCS EventArc [2] where users will consume the
> event
> > > > from
> > > > > object storage (S3 object creation for example), and then they will
> > > have
> > > > to
> > > > > add the Airflow's Create Dataset endpoint to EventBridge [3]. Now,
> if
> > > you
> > > > > just customize the payload to build the URI which is allowed
> (either
> > > via
> > > > > GET / POST) in eventbridge, it works right now. However, with the
> > > current
> > > > > proposal: a user will have to create a new user in Airflow and some
> > > > mapping
> > > > > to a function (that is either in the provider or a new user-defined
> > > > > function) that can understand this specific payload, in this
> example
> > > the
> > > > > payload for S3 events. This will become huge because this means
> that
> > > for
> > > > > each payload, we will have to provide a new function and keep it
> > > updated.
> > > > > From users POV, they will need to create a new user every time for
> a
> > > new
> > > > > service (S3, Redshift, SNS, Bedrock etc). This will again likely
> have
> > > to
> > > > go
> > > > > to the Auth manager backend. Compared to what's available today --
> i.e.
> > > > > building a URI & extra metadata that can not only work with
> EventBridge
> > > > or
> > > > > Eventarc but by any service.
> > > > >
> > > > > Since we already have to use things like EventBridge or EventArc
> for
> > > > > managed service providers to transform the event, it fits well
> with the
> > > > > existing approach. AWS Blog [3] even has a similar example for
> Datadog
> > > > > where they use input transformer "{"detail":"$.detail"}" before
> sending
> > > > it
> > > > > to Datadog's API.
> > > > > >"Having producer of an event generating even in their standard
> way,
> > > make
> > > > > it easy for airflow to consume it as a dataset event without
> external
> > > > > entities":
> > > > >
> > > > >
> > > > > [1]: https://aws.amazon.com/eventbridge/ |
> > > > >
> https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
> > > > > [2]: https://cloud.google.com/eventarc/docs
> > > > > [3]:
> > > > >
> > > > >
> > > >
> > >
> https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
> > > > >
> > > > > On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > >
> > > > > > I proposed the mapping - because it's the easiest way (I think)
> to
> > > map
> > > > > > between the "native" source to "airflow" target expectations.
> There
> > > are
> > > > > > many producers of such events, and Airflow is the consumer. And
> it
> > > > seems
> > > > > > appropriate to have a way for our users to easily plug events
> > > produced
> > > > > from
> > > > > > one system into our "events" API - without having to employ
> external
> > > > > > "mapper" (say lambda) doing the conversion. While I think it is
> > > indeed
> > > > "a
> > > > > > bit odd", it's a solution that might leverage most of what we
> have -
> > > > > > authorisation and API exposure via "user" in API.
> > > > > >
> > > > > > While I - myself - find it it a bit unusual, I think it might do
> the
> > > > job,
> > > > > > But I wonder if there is any alternative solution to the problem
> of
> > > > > "Having
> > > > > > producer of an event generating even in their standard way, make
> it
> > > > easy
> > > > > > for airflow to consume it as a dataset event without external
> > > > entities".
> > > > > >
> > > > > > On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > I would love for VOTE to get started on this one. I think most
> of
> > > the
> > > > > > > commenters and those who replied to this email are happy with
> the
> > > > > > proposal
> > > > > > > on the poll-based approach.
> > > > > > >
> > > > > > > Regarding the push-based approach, I am not convinced that the
> > > > proposed
> > > > > > > implementation has any gains over what's already available
> with the
> > > > > > Dataset
> > > > > > > Event Create API; the one user-to-one function mapping is an
> odd
> > > user
> > > > > > > experience. I'm curious to hear what others think.
> > > > > > >
> > > > > > > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > I agree with both of you that it is indeed a good idea and
> that
> > > it
> > > > > can
> > > > > > be
> > > > > > > > added in Future work -- doesn't need to be part of this AIP.
> > > > > > > >
> > > > > > > > Thanks for the interest. I was not aware of such feature and
> this
> > > > > looks
> > > > > > > >> really cool! I definitely think that can be useful for
> Airflow,
> > > > > > > especially
> > > > > > > >> for testing when you can easily replay events received in
> the
> > > > past.
> > > > > > > >> However, I do not think it should be part of the AIP and,
> as you
> > > > > > > mentioned,
> > > > > > > >> if should be a future work or a follow-up item of the AIP.
> > > Please
> > > > > let
> > > > > > me
> > > > > > > >> know if you (or anyone) disagree with this and we can talk
> about
> > > > it.
> > > > > > > >> Otherwise I'll update the future work section of the AIP and
> > > > mention
> > > > > > > this
> > > > > > > >> archive and replay feature.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <
> vincbeck@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> Hey Pavan,
> > > > > > > >>
> > > > > > > >> Thanks for the interest. I was not aware of such feature and
> > > this
> > > > > > looks
> > > > > > > >> really cool! I definitely think that can be useful for
> Airflow,
> > > > > > > especially
> > > > > > > >> for testing when you can easily replay events received in
> the
> > > > past.
> > > > > > > >> However, I do not think it should be part of the AIP and,
> as you
> > > > > > > mentioned,
> > > > > > > >> if should be a future work or a follow-up item of the AIP.
> > > Please
> > > > > let
> > > > > > me
> > > > > > > >> know if you (or anyone) disagree with this and we can talk
> about
> > > > it.
> > > > > > > >> Otherwise I'll update the future work section of the AIP and
> > > > mention
> > > > > > > this
> > > > > > > >> archive and replay feature.
> > > > > > > >>
> > > > > > > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > > > > > > >> > Thanks Vincent, I took a look , this is really good. Don't
> > > have
> > > > > > access
> > > > > > > >> to
> > > > > > > >> > the confluence page to comment :) so adding it here.
> > > > > > > >> >
> > > > > > > >> > As events arrive-->do somework-->end.
> > > > > > > >> >
> > > > > > > >> > So I'm uncertain if my comment pertains to the current
> > > poll/push
> > > > > > model
> > > > > > > >> or
> > > > > > > >> > if it fits part of future work(seen event batching ).
> > > > > > > >> >
> > > > > > > >> > Have you given any thought to the event archival
> mechanism and
> > > > > event
> > > > > > > >> > replay? This could significantly aid in testing and
> recovery
> > > of
> > > > > > > workflow
> > > > > > > >> > and testing new functionality with events by just replay
> the
> > > > > events.
> > > > > > > The
> > > > > > > >> > archival mechanism I am referring to is similar to today
> in
> > > AWS
> > > > we
> > > > > > > have
> > > > > > > >> > Event Bridge Archive and Replay.
> > > > > > > >> >
> > > > > > > >> > Regards,
> > > > > > > >> > Pavan
> > > > > > > >> >
> > > > > > > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > I actually did manage to take a look, thanks for the
> work. I
> > > > am
> > > > > +1
> > > > > > > on
> > > > > > > >> the
> > > > > > > >> > > poll-based approach -- left a comment on the
> push-based: I
> > > am
> > > > > not
> > > > > > > >> sure of
> > > > > > > >> > > why we need a function since create asset event API
> endpoint
> > > > > > should
> > > > > > > >> have
> > > > > > > >> > > all info needed for what the Asset was.
> > > > > > > >> > >
> > > > > > > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <
> > > kaxilnaik@gmail.com>
> > > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <
> > > > > vincbeck@apache.org
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >> > > >
> > > > > > > >> > > >> Hi everyone,
> > > > > > > >> > > >>
> > > > > > > >> > > >> I updated the AIP-82 given the different comments and
> > > > > concerns
> > > > > > I
> > > > > > > >> > > >> received. I also tried to reply to all comments
> > > > > individually. I
> > > > > > > >> would
> > > > > > > >> > > >> really appreciate if you can do a second pass and
> let me
> > > > know
> > > > > > > what
> > > > > > > >> you
> > > > > > > >> > > >> think. Overall, this is what I changed in the AIP:
> > > > > > > >> > > >>
> > > > > > > >> > > >> - Push based event-driven scheduling. I updated this
> > > > section
> > > > > > > >> entirely
> > > > > > > >> > > >> because I received many concerns about the previous
> > > > proposal.
> > > > > > The
> > > > > > > >> > > overall
> > > > > > > >> > > >> idea now is to leverage the create asset event API
> > > endpoint
> > > > > to
> > > > > > > send
> > > > > > > >> > > >> notifications from external (e.g. cloud provider) to
> > > > Airflow
> > > > > > > >> > > environment.
> > > > > > > >> > > >>
> > > > > > > >> > > >> - I updated the poll based event-driven scheduling
> DAG
> > > > author
> > > > > > > >> experience
> > > > > > > >> > > >> to use a message queue scenario. I understood that
> this
> > > is
> > > > > > > >> probably the
> > > > > > > >> > > >> main use case we are trying to cover with this AIP,
> thus
> > > I
> > > > > used
> > > > > > > it
> > > > > > > >> as
> > > > > > > >> > > >> example and mentioned it multiple times across the
> AIP.
> > > > > > > >> > > >>
> > > > > > > >> > > >> Thanks again for your time :)
> > > > > > > >> > > >>
> > > > > > > >> > > >>
> > > > > > > >> > > >>
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > > > > >> > > >>
> > > > > > > >> > > >> Vincent
> > > > > > > >> > > >>
> > > > > > > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > > > > > >> > > >> > Thanks a lot all for the comments, this is very
> much
> > > > > > > >> appreciated! I
> > > > > > > >> > > >> received many comments from this thread and in
> > > confluence,
> > > > > > thanks
> > > > > > > >> again.
> > > > > > > >> > > >> I'll try to address them all in the AIP and will
> send an
> > > > > email
> > > > > > in
> > > > > > > >> this
> > > > > > > >> > > >> thread once done. I will most likely revisit the
> > > push-based
> > > > > > > >> approach
> > > > > > > >> > > given
> > > > > > > >> > > >> the number of concerns I received, thanks Jarek for
> > > > proposing
> > > > > > > >> another
> > > > > > > >> > > >> solution, I'll probably go down that path.
> > > > > > > >> > > >> >
> > > > > > > >> > > >> > One follow-up question Vikram.
> > > > > > > >> > > >> >
> > > > > > > >> > > >> > > The bespoke triggerer approach completely makes
> sense
> > > > for
> > > > > > the
> > > > > > > >> long
> > > > > > > >> > > >> tail here, but can we do better for the 20% of
> scenarios
> > > > > which
> > > > > > > >> cover
> > > > > > > >> > > well
> > > > > > > >> > > >> over 80% of usage here is the question in my mind.
> Or,
> > > are
> > > > > you
> > > > > > > >> thinking
> > > > > > > >> > > of
> > > > > > > >> > > >> those as being covered in the "push" model?
> > > > > > > >> > > >> >
> > > > > > > >> > > >> > Could you share more details about what is this
> "20% of
> > > > > > > scenarios
> > > > > > > >> > > which
> > > > > > > >> > > >> cover well over 80% of usage" please?
> > > > > > > >> > > >> >
> > > > > > > >> > > >> > Vincent
> > > > > > > >> > > >> >
> > > > > > > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > > > > > >> > > >> > > Thanks Vincent for driving these, I have added my
> > > > > comments
> > > > > > to
> > > > > > > >> the
> > > > > > > >> > > AIP
> > > > > > > >> > > >> too.
> > > > > > > >> > > >> > >
> > > > > > > >> > > >> > > Regards,
> > > > > > > >> > > >> > > Kaxil
> > > > > > > >> > > >> > >
> > > > > > > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > > > > > > (XC-AS/EAE-ADA-T)
> > > > > > > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > > > > > >> > > >> > >
> > > > > > > >> > > >> > > > +1 on the comments of Vikram and Jarek, added
> main
> > > > > points
> > > > > > > on
> > > > > > > >> > > >> confluence
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > Sent from Outlook for iOS<
> https://aka.ms/o0ukef>
> > > > > > > >> > > >> > > > ________________________________
> > > > > > > >> > > >> > > > From: Vikram Koka <vikram@astronomer.io.INVALID
> >
> > > > > > > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > > > > > >> > > >> > > > To: dev@airflow.apache.org <
> dev@airflow.apache.org
> > > >
> > > > > > > >> > > >> > > > Subject: Re: [DISCUSS] External event driven
> > > > scheduling
> > > > > > in
> > > > > > > >> Airflow
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > Vincent,
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > Thanks for writing this up. The overview looks
> > > really
> > > > > > good!
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > I will leave my comments in the AIP as well,
> but
> > > at a
> > > > > > high
> > > > > > > >> level
> > > > > > > >> > > >> they are
> > > > > > > >> > > >> > > > both relatively focused on the "how", rather
> than
> > > the
> > > > > > > "what".
> > > > > > > >> > > >> > > > With respect to the pull / polling approach, I
> > > > > completely
> > > > > > > >> agree
> > > > > > > >> > > >> that some
> > > > > > > >> > > >> > > > incarnation of this is needed.
> > > > > > > >> > > >> > > > I am less certain as to how on this part. The
> > > bespoke
> > > > > > > >> triggerer
> > > > > > > >> > > >> approach
> > > > > > > >> > > >> > > > completely makes sense for the long tail here,
> but
> > > > can
> > > > > we
> > > > > > > do
> > > > > > > >> > > better
> > > > > > > >> > > >> for the
> > > > > > > >> > > >> > > > 20% of scenarios which cover well over 80% of
> usage
> > > > > here
> > > > > > is
> > > > > > > >> the
> > > > > > > >> > > >> question in
> > > > > > > >> > > >> > > > my mind. Or, are you thinking of those as being
> > > > covered
> > > > > > in
> > > > > > > >> the
> > > > > > > >> > > >> "push"
> > > > > > > >> > > >> > > > model?
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > Which leads to the "push" model approach.
> > > > > > > >> > > >> > > > I am struggling with the same question that
> Jarek
> > > > > raised
> > > > > > > here
> > > > > > > >> > > about
> > > > > > > >> > > >> whether
> > > > > > > >> > > >> > > > we need a new Airflow entity over and beyond
> the
> > > > > existing
> > > > > > > >> REST API
> > > > > > > >> > > >> for the
> > > > > > > >> > > >> > > > same.
> > > > > > > >> > > >> > > > I am concerned about this becoming a vector of
> > > attack
> > > > > on
> > > > > > > >> Airflow.
> > > > > > > >> > > >> > > > I see that this is a hot topic of discussion
> in the
> > > > > > > >> Confluence doc
> > > > > > > >> > > >> as well,
> > > > > > > >> > > >> > > > but wanted to summarize here as well, so it
> didn't
> > > > get
> > > > > > lost
> > > > > > > >> in the
> > > > > > > >> > > >> threads
> > > > > > > >> > > >> > > > of comments.
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > Best regards,
> > > > > > > >> > > >> > > > Vikram
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > > > > > > >> jarek@potiuk.com>
> > > > > > > >> > > >> wrote:
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > > > > Thanks Vincent. I took a look and I have a
> > > general
> > > > > > > >> comment. I
> > > > > > > >> > > >> > > > > strongly think external driven scheduling is
> > > really
> > > > > > > needed
> > > > > > > >> -
> > > > > > > >> > > >> especially,
> > > > > > > >> > > >> > > > it
> > > > > > > >> > > >> > > > > should be much easier for a user to "plug-in"
> > > such
> > > > an
> > > > > > > >> external
> > > > > > > >> > > >> event to
> > > > > > > >> > > >> > > > > Airflow. And there are two parts of it - as
> > > > correctly
> > > > > > > >> stated
> > > > > > > >> > > >> there - pull
> > > > > > > >> > > >> > > > > and push.
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > For the pull - I think it would be great to
> have
> > > a
> > > > > kind
> > > > > > > of
> > > > > > > >> > > >> specialized
> > > > > > > >> > > >> > > > > Triggers that will be started when DAG is
> parsed
> > > -
> > > > > and
> > > > > > > >> those
> > > > > > > >> > > >> Triggers
> > > > > > > >> > > >> > > > could
> > > > > > > >> > > >> > > > > generate the events for DAGs. I think
> basically
> > > > > that's
> > > > > > > all
> > > > > > > >> that
> > > > > > > >> > > is
> > > > > > > >> > > >> > > > needed,
> > > > > > > >> > > >> > > > > for example I imagine a pubsub trigger that
> will
> > > > > > > subscribe
> > > > > > > >> to
> > > > > > > >> > > >> messages
> > > > > > > >> > > >> > > > > coming on the pubsub queue and fire "Asset"
> event
> > > > > when
> > > > > > a
> > > > > > > >> message
> > > > > > > >> > > >> is
> > > > > > > >> > > >> > > > > received. Not much controversy there - I am
> not
> > > > sure
> > > > > > > about
> > > > > > > >> the
> > > > > > > >> > > >> polling
> > > > > > > >> > > >> > > > > thing , because I've always believed that
> when
> > > > > > > >> "asyncio-native"
> > > > > > > >> > > >> Trigger
> > > > > > > >> > > >> > > > is
> > > > > > > >> > > >> > > > > run in the asyncio event loop, we do not
> "poll"
> > > > every
> > > > > > > >> second or
> > > > > > > >> > > >> so (but
> > > > > > > >> > > >> > > > > maybe this is just coming from some specific
> > > > triggers
> > > > > > > that
> > > > > > > >> > > >> actually do
> > > > > > > >> > > >> > > > > such regular poll. But yes - there are polls
> > > like
> > > > > > > running
> > > > > > > >> > > select
> > > > > > > >> > > >> on the
> > > > > > > >> > > >> > > > DB
> > > > > > > >> > > >> > > > > that cannot be easily "async-ed" so having a
> > > > > > configurable
> > > > > > > >> > > polling
> > > > > > > >> > > >> time
> > > > > > > >> > > >> > > > > would be good there (but I am not sure maybe
> it's
> > > > > even
> > > > > > > >> possible
> > > > > > > >> > > >> today). I
> > > > > > > >> > > >> > > > > think this would be really great if we have
> that
> > > > > > option,
> > > > > > > >> because
> > > > > > > >> > > >> it makes
> > > > > > > >> > > >> > > > > it much easier to set up the authorization
> for
> > > > > Airlfow
> > > > > > > >> users -
> > > > > > > >> > > >> rather
> > > > > > > >> > > >> > > > than
> > > > > > > >> > > >> > > > > setting up authorization and REST calls
> coming
> > > from
> > > > > an
> > > > > > > >> external
> > > > > > > >> > > >> system,
> > > > > > > >> > > >> > > > we
> > > > > > > >> > > >> > > > > can utilize Connections of Airlfow to
> authorize
> > > > such
> > > > > a
> > > > > > > >> Trigger
> > > > > > > >> > > to
> > > > > > > >> > > >> > > > subscribe
> > > > > > > >> > > >> > > > > to events.
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > For the push proposal - as I read the
> proposal,
> > > > the
> > > > > > main
> > > > > > > >> point
> > > > > > > >> > > >> behind it
> > > > > > > >> > > >> > > > > is rather than users having to write
> "Airflow"
> > > way
> > > > of
> > > > > > > >> triggering
> > > > > > > >> > > >> events
> > > > > > > >> > > >> > > > and
> > > > > > > >> > > >> > > > > configuring authentication (using REST API)
> to
> > > > > generate
> > > > > > > >> asset
> > > > > > > >> > > >> events, is
> > > > > > > >> > > >> > > > to
> > > > > > > >> > > >> > > > > make Airflow natively understand external
> ways of
> > > > > > pushing
> > > > > > > >> - and
> > > > > > > >> > > >> > > > effectively
> > > > > > > >> > > >> > > > > authorizing and mapping such incoming
> > > unauthorized
> > > > > > > >> requests into
> > > > > > > >> > > >> event
> > > > > > > >> > > >> > > > that
> > > > > > > >> > > >> > > > > could be generated by an API REST call.
> > > > > > > >> > > >> > > > > I am not really sure honestly if this is
> > > something
> > > > > that
> > > > > > > we
> > > > > > > >> want
> > > > > > > >> > > as
> > > > > > > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say
> such
> > > > an
> > > > > > > >> > > unauthorised
> > > > > > > >> > > >> > > > endpoint
> > > > > > > >> > > >> > > > > is probably not a good idea - for a variety
> of
> > > > > reasons,
> > > > > > > >> mostly
> > > > > > > >> > > >> security.
> > > > > > > >> > > >> > > > > And as I understand the goal is that users
> can
> > > > easily
> > > > > > > >> point at
> > > > > > > >> > > >> > > > "3rd-party"
> > > > > > > >> > > >> > > > > notification to Airflow and get the event
> > > > generated.
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > My feeling is that while this is needed - it
> > > should
> > > > > be
> > > > > > > >> > > >> externalised from
> > > > > > > >> > > >> > > > > airlfow webserver. The authorization has to
> be
> > > set
> > > > up
> > > > > > > >> anyway
> > > > > > > >> > > >> > > > additionally -
> > > > > > > >> > > >> > > > > unlike in "poll" case - we cannot use
> Connections
> > > > for
> > > > > > > >> > > authorizing
> > > > > > > >> > > >> > > > (because
> > > > > > > >> > > >> > > > > it's not Airlfow that authorizes in an
> external
> > > > > system
> > > > > > -
> > > > > > > >> it's
> > > > > > > >> > > the
> > > > > > > >> > > >> other
> > > > > > > >> > > >> > > > way
> > > > > > > >> > > >> > > > > round). So we have to anyhow setup "something
> > > > extra"
> > > > > in
> > > > > > > >> Airflow
> > > > > > > >> > > to
> > > > > > > >> > > >> > > > > authorize the external system. Which could be
> > > what
> > > > we
> > > > > > > have
> > > > > > > >> now -
> > > > > > > >> > > >> user
> > > > > > > >> > > >> > > > that
> > > > > > > >> > > >> > > > > allows us to trigger the event. Which means
> that
> > > > our
> > > > > > REST
> > > > > > > >> API
> > > > > > > >> > > >> could
> > > > > > > >> > > >> > > > > potentially be used the same way it is now,
> but
> > > we
> > > > > will
> > > > > > > >> need
> > > > > > > >> > > >> "something"
> > > > > > > >> > > >> > > > > (library, lambda function etc.) that users
> could
> > > > > easily
> > > > > > > >> setup in
> > > > > > > >> > > >> the
> > > > > > > >> > > >> > > > > external system to map whatever trigger they
> > > > generate
> > > > > > > >> natively
> > > > > > > >> > > >> (say S3
> > > > > > > >> > > >> > > > file
> > > > > > > >> > > >> > > > > created) to Airflow REST API.
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > As I see it - this is quite often used (and
> very
> > > > > > > >> practical, that
> > > > > > > >> > > >> you
> > > > > > > >> > > >> > > > deploy
> > > > > > > >> > > >> > > > > a cloud function or lambda that subscribes
> on the
> > > > > event
> > > > > > > >> received
> > > > > > > >> > > >> when
> > > > > > > >> > > >> > > > > S3/GCS is created. So it would be on the
> user to
> > > > > deploy
> > > > > > > >> such a
> > > > > > > >> > > >> lambda -
> > > > > > > >> > > >> > > > but
> > > > > > > >> > > >> > > > > we **could** provide a library of those: say
> s3
> > > > > lambda,
> > > > > > > gcp
> > > > > > > >> > > cloud
> > > > > > > >> > > >> > > > function
> > > > > > > >> > > >> > > > > in respective providers - with documentation
> how
> > > to
> > > > > set
> > > > > > > >> them up,
> > > > > > > >> > > >> and how
> > > > > > > >> > > >> > > > to
> > > > > > > >> > > >> > > > > configure authorization and we would be
> generally
> > > > > > "done".
> > > > > > > >> I am
> > > > > > > >> > > >> just not
> > > > > > > >> > > >> > > > > sure if we need a new entity in Airflow for
> that
> > > > > (Event
> > > > > > > >> > > >> receiver). It
> > > > > > > >> > > >> > > > feels
> > > > > > > >> > > >> > > > > like it asks Airflow to take more
> responsibility,
> > > > > when
> > > > > > we
> > > > > > > >> all
> > > > > > > >> > > >> think on
> > > > > > > >> > > >> > > > what
> > > > > > > >> > > >> > > > > to "remove" from Airflow rather than "add"
> to it
> > > -
> > > > > > > >> especially
> > > > > > > >> > > >> when it
> > > > > > > >> > > >> > > > comes
> > > > > > > >> > > >> > > > > to external integrations. It feels to me that
> > > > Airflow
> > > > > > > >> should
> > > > > > > >> > > make
> > > > > > > >> > > >> it easy
> > > > > > > >> > > >> > > > > to be triggered by such an external system
> and
> > > make
> > > > > it
> > > > > > > >> easy to
> > > > > > > >> > > >> "map" to
> > > > > > > >> > > >> > > > the
> > > > > > > >> > > >> > > > > way we expect to get events triggered, but
> this
> > > > > should
> > > > > > be
> > > > > > > >> done
> > > > > > > >> > > >> outside of
> > > > > > > >> > > >> > > > > Airflow. If the users can easily find in our
> docs
> > > > > when
> > > > > > > they
> > > > > > > >> > > >> search "what
> > > > > > > >> > > >> > > > do
> > > > > > > >> > > >> > > > > I do to externally trigger Airflow on S3
> change":
> > > > > > either
> > > > > > > a)
> > > > > > > >> > > >> configure
> > > > > > > >> > > >> > > > > polling in airflow using s3 Connection, or b)
> > > > > "create a
> > > > > > > >> user +
> > > > > > > >> > > >> deploy
> > > > > > > >> > > >> > > > this
> > > > > > > >> > > >> > > > > lambda with those parameters" - that is
> "easy
> > > > > enough"
> > > > > > > and
> > > > > > > >> very
> > > > > > > >> > > >> practical
> > > > > > > >> > > >> > > > > as well.
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > But maybe I am not seeing the whole picture
> and
> > > the
> > > > > > real
> > > > > > > >> problem
> > > > > > > >> > > >> it's
> > > > > > > >> > > >> > > > > solving - so take it as a "first review
> pass" and
> > > > > "guts
> > > > > > > >> > > feeling".
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > J.
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck,
> Vincent
> > > > > > > >> > > >> > > > <vincbeck@amazon.com.invalid
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > > wrote:
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > > Hello everyone,
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > > > I created a draft AIP regarding "External
> event
> > > > > > driven
> > > > > > > >> > > >> scheduling in
> > > > > > > >> > > >> > > > > > Airflow". This proposal is about adding
> > > > capability
> > > > > in
> > > > > > > >> Airflow
> > > > > > > >> > > to
> > > > > > > >> > > >> > > > schedule
> > > > > > > >> > > >> > > > > > DAGs based on external events. Here are
> some
> > > > > examples
> > > > > > > of
> > > > > > > >> such
> > > > > > > >> > > >> external
> > > > > > > >> > > >> > > > > > events:
> > > > > > > >> > > >> > > > > > - A user signs up to one of the user pool
> > > defined
> > > > > in
> > > > > > my
> > > > > > > >> cloud
> > > > > > > >> > > >> provider
> > > > > > > >> > > >> > > > > > - One of the databases used in my company
> has
> > > > been
> > > > > > > >> updated
> > > > > > > >> > > >> > > > > > - A job in my cloud provider has been
> executed
> > > > > > > >> successfully
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > > > The intent of this AIP is to leverage
> datasets
> > > > > (which
> > > > > > > >> will be
> > > > > > > >> > > >> soon
> > > > > > > >> > > >> > > > > assets)
> > > > > > > >> > > >> > > > > > and update them based on external events. I
> > > would
> > > > > > like
> > > > > > > to
> > > > > > > >> > > >> propose this
> > > > > > > >> > > >> > > > > AIP
> > > > > > > >> > > >> > > > > > for discussion and more importantly, hear
> some
> > > > > > > feedbacks
> > > > > > > >> from
> > > > > > > >> > > >> you :)
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >>
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > > > > > >> > > >> > > > <
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >>
> > > > > > > >> > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > > > Vincent
> > > > > > > >> > > >> > > > > >
> > > > > > > >> > > >> > > > >
> > > > > > > >> > > >> > > >
> > > > > > > >> > > >> > >
> > > > > > > >> > > >> >
> > > > > > > >> > > >> >
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >> > > >> > To unsubscribe, e-mail:
> > > > dev-unsubscribe@airflow.apache.org
> > > > > > > >> > > >> > For additional commands, e-mail:
> > > > > dev-help@airflow.apache.org
> > > > > > > >> > > >> >
> > > > > > > >> > > >> >
> > > > > > > >> > > >>
> > > > > > > >> > > >>
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >> > > >> To unsubscribe, e-mail:
> > > dev-unsubscribe@airflow.apache.org
> > > > > > > >> > > >> For additional commands, e-mail:
> > > > dev-help@airflow.apache.org
> > > > > > > >> > > >>
> > > > > > > >> > > >>
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > >
> ---------------------------------------------------------------------
> > > > > > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > >> For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vincent Beck <vi...@apache.org>.
Hello everyone,
After multiple conversations with different actors, I decided to move the push based event scheduling out of scope of this AIP. I could not find a solution for the push based approach that was satisfactory and I think more investigation and data from Airflow users is needed to move on with this approach. In order to not block the poll based event scheduling, which did not receive any concern, I decided to focus AIP-82 on the poll-based event scheduling. I updated the AIP accordingly.
Since the poll-based approach did not receive any concerns (and many positive feedbacks), I'll start voting in another thread.
Thanks for your feedbacks!
Vincent
On 2024/08/02 00:50:35 Pavankumar Gopidesu wrote:
> Thanks Vincent and Kaxil, Agree it can be added to future work.
>
> Looking at discussion on event mapping, Today we know the source systems
> and we are aware of the schemas from source systems (S3, SQS, Event Arc
> etc;)
>
> However, as new services emerge and their schemas evolve, this may change.
>
> Agree with Kaxil has mentioned that transformers are very effective in
> converting the required format before sending data to target systems.
>
> Is it a good idea to define a schema for events to enforce from airflow?
> There are some open standards, such as CloudEvents[1], that we could
> consider.
>
> This approach would allow us to maintain versions (v1, v2, ... vn) as we
> evolve, providing utilities that users can easily plug into and send events
> to Airflow.
>
> It would also simplify maintenance within the Airflow system to handle
> incoming events because we get a known version of the schema from
> user request.
>
>
> If this is already being addressed, please disregard my comments.
>
>
> [1]: https://github.com/cloudevents/spec
>
>
> Regards,
>
> Pavan
>
> On Fri, Aug 2, 2024 at 1:40 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Yes but the big difference is you will create a single user for EventBridge
> > since that is the one sending request to Airflow, single user for EventArc
> > for GCP and 1 user for every other EventListener or application ---- as
> > compared to 1 user per type of Payload (since Airflow will need to
> > understand the payload of the original source). So in that case, you will
> > have 1 user,function mapping for S3, 1user,function mapping for Redshift,
> > and it goes on. The former approach is also consistent with our Connection
> > model, where we have one standardized AWS & GCP connection that works for
> > most, if not all, services.
> >
> > but the user will have to be added anyway (some kind of
> > > service account - because the API needs to be authorized - that part is
> > not
> > > changed). (unless of course you want to use the same user for all kinds
> > of
> > > external interfaces, which for security point of view is a very bad idea
> > -
> > > each external system should have their own "service account" - that 's
> > the
> > > best practices from the security point of view.
> >
> >
> > On Fri, 2 Aug 2024 at 01:22, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I am all for it - if we want to stick to event bridge or similar and
> > > recommend it to our users, it's perfectly fine for me, It would be great
> > > however to add a documentation explaining the steps and some examples -
> > > ideally for most of our providers and "standard" ways of triggering such
> > an
> > > event. This is even what I proposed originally whent the first version of
> > > the document was created ( to just document how to map the events
> > > externally).
> > >
> > > BTW. Yes - in this case you need to implement the logic in the
> > > event bridge, but the user will have to be added anyway (some kind of
> > > service account - because the API needs to be authorized - that part is
> > not
> > > changed). (unless of course you want to use the same user for all kinds
> > of
> > > external interfaces, which for security point of view is a very bad idea
> > -
> > > each external system should have their own "service account" - that 's
> > the
> > > best practices from the security point of view.
> > >
> > > J,
> > >
> > > On Fri, Aug 2, 2024 at 2:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > I was discussing this with Vincent. In either case, same as now or the
> > > one
> > > > proposed in the AIP, a user will have to use something like AWS
> > > > EventBridge[1] or GCS EventArc [2] where users will consume the event
> > > from
> > > > object storage (S3 object creation for example), and then they will
> > have
> > > to
> > > > add the Airflow's Create Dataset endpoint to EventBridge [3]. Now, if
> > you
> > > > just customize the payload to build the URI which is allowed (either
> > via
> > > > GET / POST) in eventbridge, it works right now. However, with the
> > current
> > > > proposal: a user will have to create a new user in Airflow and some
> > > mapping
> > > > to a function (that is either in the provider or a new user-defined
> > > > function) that can understand this specific payload, in this example
> > the
> > > > payload for S3 events. This will become huge because this means that
> > for
> > > > each payload, we will have to provide a new function and keep it
> > updated.
> > > > From users POV, they will need to create a new user every time for a
> > new
> > > > service (S3, Redshift, SNS, Bedrock etc). This will again likely have
> > to
> > > go
> > > > to the Auth manager backend. Compared to what's available today -- i.e.
> > > > building a URI & extra metadata that can not only work with EventBridge
> > > or
> > > > Eventarc but by any service.
> > > >
> > > > Since we already have to use things like EventBridge or EventArc for
> > > > managed service providers to transform the event, it fits well with the
> > > > existing approach. AWS Blog [3] even has a similar example for Datadog
> > > > where they use input transformer "{"detail":"$.detail"}" before sending
> > > it
> > > > to Datadog's API.
> > > > >"Having producer of an event generating even in their standard way,
> > make
> > > > it easy for airflow to consume it as a dataset event without external
> > > > entities":
> > > >
> > > >
> > > > [1]: https://aws.amazon.com/eventbridge/ |
> > > > https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
> > > > [2]: https://cloud.google.com/eventarc/docs
> > > > [3]:
> > > >
> > > >
> > >
> > https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
> > > >
> > > > On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > > > I proposed the mapping - because it's the easiest way (I think) to
> > map
> > > > > between the "native" source to "airflow" target expectations. There
> > are
> > > > > many producers of such events, and Airflow is the consumer. And it
> > > seems
> > > > > appropriate to have a way for our users to easily plug events
> > produced
> > > > from
> > > > > one system into our "events" API - without having to employ external
> > > > > "mapper" (say lambda) doing the conversion. While I think it is
> > indeed
> > > "a
> > > > > bit odd", it's a solution that might leverage most of what we have -
> > > > > authorisation and API exposure via "user" in API.
> > > > >
> > > > > While I - myself - find it it a bit unusual, I think it might do the
> > > job,
> > > > > But I wonder if there is any alternative solution to the problem of
> > > > "Having
> > > > > producer of an event generating even in their standard way, make it
> > > easy
> > > > > for airflow to consume it as a dataset event without external
> > > entities".
> > > > >
> > > > > On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I would love for VOTE to get started on this one. I think most of
> > the
> > > > > > commenters and those who replied to this email are happy with the
> > > > > proposal
> > > > > > on the poll-based approach.
> > > > > >
> > > > > > Regarding the push-based approach, I am not convinced that the
> > > proposed
> > > > > > implementation has any gains over what's already available with the
> > > > > Dataset
> > > > > > Event Create API; the one user-to-one function mapping is an odd
> > user
> > > > > > experience. I'm curious to hear what others think.
> > > > > >
> > > > > > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > I agree with both of you that it is indeed a good idea and that
> > it
> > > > can
> > > > > be
> > > > > > > added in Future work -- doesn't need to be part of this AIP.
> > > > > > >
> > > > > > > Thanks for the interest. I was not aware of such feature and this
> > > > looks
> > > > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > > > especially
> > > > > > >> for testing when you can easily replay events received in the
> > > past.
> > > > > > >> However, I do not think it should be part of the AIP and, as you
> > > > > > mentioned,
> > > > > > >> if should be a future work or a follow-up item of the AIP.
> > Please
> > > > let
> > > > > me
> > > > > > >> know if you (or anyone) disagree with this and we can talk about
> > > it.
> > > > > > >> Otherwise I'll update the future work section of the AIP and
> > > mention
> > > > > > this
> > > > > > >> archive and replay feature.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > >> Hey Pavan,
> > > > > > >>
> > > > > > >> Thanks for the interest. I was not aware of such feature and
> > this
> > > > > looks
> > > > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > > > especially
> > > > > > >> for testing when you can easily replay events received in the
> > > past.
> > > > > > >> However, I do not think it should be part of the AIP and, as you
> > > > > > mentioned,
> > > > > > >> if should be a future work or a follow-up item of the AIP.
> > Please
> > > > let
> > > > > me
> > > > > > >> know if you (or anyone) disagree with this and we can talk about
> > > it.
> > > > > > >> Otherwise I'll update the future work section of the AIP and
> > > mention
> > > > > > this
> > > > > > >> archive and replay feature.
> > > > > > >>
> > > > > > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > > > > > >> > Thanks Vincent, I took a look , this is really good. Don't
> > have
> > > > > access
> > > > > > >> to
> > > > > > >> > the confluence page to comment :) so adding it here.
> > > > > > >> >
> > > > > > >> > As events arrive-->do somework-->end.
> > > > > > >> >
> > > > > > >> > So I'm uncertain if my comment pertains to the current
> > poll/push
> > > > > model
> > > > > > >> or
> > > > > > >> > if it fits part of future work(seen event batching ).
> > > > > > >> >
> > > > > > >> > Have you given any thought to the event archival mechanism and
> > > > event
> > > > > > >> > replay? This could significantly aid in testing and recovery
> > of
> > > > > > workflow
> > > > > > >> > and testing new functionality with events by just replay the
> > > > events.
> > > > > > The
> > > > > > >> > archival mechanism I am referring to is similar to today in
> > AWS
> > > we
> > > > > > have
> > > > > > >> > Event Bridge Archive and Replay.
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> > Pavan
> > > > > > >> >
> > > > > > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <
> > kaxilnaik@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > I actually did manage to take a look, thanks for the work. I
> > > am
> > > > +1
> > > > > > on
> > > > > > >> the
> > > > > > >> > > poll-based approach -- left a comment on the push-based: I
> > am
> > > > not
> > > > > > >> sure of
> > > > > > >> > > why we need a function since create asset event API endpoint
> > > > > should
> > > > > > >> have
> > > > > > >> > > all info needed for what the Asset was.
> > > > > > >> > >
> > > > > > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <
> > kaxilnaik@gmail.com>
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > > > > > >> > > >
> > > > > > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <
> > > > vincbeck@apache.org
> > > > > >
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > >> Hi everyone,
> > > > > > >> > > >>
> > > > > > >> > > >> I updated the AIP-82 given the different comments and
> > > > concerns
> > > > > I
> > > > > > >> > > >> received. I also tried to reply to all comments
> > > > individually. I
> > > > > > >> would
> > > > > > >> > > >> really appreciate if you can do a second pass and let me
> > > know
> > > > > > what
> > > > > > >> you
> > > > > > >> > > >> think. Overall, this is what I changed in the AIP:
> > > > > > >> > > >>
> > > > > > >> > > >> - Push based event-driven scheduling. I updated this
> > > section
> > > > > > >> entirely
> > > > > > >> > > >> because I received many concerns about the previous
> > > proposal.
> > > > > The
> > > > > > >> > > overall
> > > > > > >> > > >> idea now is to leverage the create asset event API
> > endpoint
> > > > to
> > > > > > send
> > > > > > >> > > >> notifications from external (e.g. cloud provider) to
> > > Airflow
> > > > > > >> > > environment.
> > > > > > >> > > >>
> > > > > > >> > > >> - I updated the poll based event-driven scheduling DAG
> > > author
> > > > > > >> experience
> > > > > > >> > > >> to use a message queue scenario. I understood that this
> > is
> > > > > > >> probably the
> > > > > > >> > > >> main use case we are trying to cover with this AIP, thus
> > I
> > > > used
> > > > > > it
> > > > > > >> as
> > > > > > >> > > >> example and mentioned it multiple times across the AIP.
> > > > > > >> > > >>
> > > > > > >> > > >> Thanks again for your time :)
> > > > > > >> > > >>
> > > > > > >> > > >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > > > >> > > >>
> > > > > > >> > > >> Vincent
> > > > > > >> > > >>
> > > > > > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > > > > >> > > >> > Thanks a lot all for the comments, this is very much
> > > > > > >> appreciated! I
> > > > > > >> > > >> received many comments from this thread and in
> > confluence,
> > > > > thanks
> > > > > > >> again.
> > > > > > >> > > >> I'll try to address them all in the AIP and will send an
> > > > email
> > > > > in
> > > > > > >> this
> > > > > > >> > > >> thread once done. I will most likely revisit the
> > push-based
> > > > > > >> approach
> > > > > > >> > > given
> > > > > > >> > > >> the number of concerns I received, thanks Jarek for
> > > proposing
> > > > > > >> another
> > > > > > >> > > >> solution, I'll probably go down that path.
> > > > > > >> > > >> >
> > > > > > >> > > >> > One follow-up question Vikram.
> > > > > > >> > > >> >
> > > > > > >> > > >> > > The bespoke triggerer approach completely makes sense
> > > for
> > > > > the
> > > > > > >> long
> > > > > > >> > > >> tail here, but can we do better for the 20% of scenarios
> > > > which
> > > > > > >> cover
> > > > > > >> > > well
> > > > > > >> > > >> over 80% of usage here is the question in my mind. Or,
> > are
> > > > you
> > > > > > >> thinking
> > > > > > >> > > of
> > > > > > >> > > >> those as being covered in the "push" model?
> > > > > > >> > > >> >
> > > > > > >> > > >> > Could you share more details about what is this "20% of
> > > > > > scenarios
> > > > > > >> > > which
> > > > > > >> > > >> cover well over 80% of usage" please?
> > > > > > >> > > >> >
> > > > > > >> > > >> > Vincent
> > > > > > >> > > >> >
> > > > > > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > > > > >> > > >> > > Thanks Vincent for driving these, I have added my
> > > > comments
> > > > > to
> > > > > > >> the
> > > > > > >> > > AIP
> > > > > > >> > > >> too.
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > Regards,
> > > > > > >> > > >> > > Kaxil
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > > > > > (XC-AS/EAE-ADA-T)
> > > > > > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > > +1 on the comments of Vikram and Jarek, added main
> > > > points
> > > > > > on
> > > > > > >> > > >> confluence
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > > > > >> > > >> > > > ________________________________
> > > > > > >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > > > > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > > > > >> > > >> > > > To: dev@airflow.apache.org <dev@airflow.apache.org
> > >
> > > > > > >> > > >> > > > Subject: Re: [DISCUSS] External event driven
> > > scheduling
> > > > > in
> > > > > > >> Airflow
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Vincent,
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Thanks for writing this up. The overview looks
> > really
> > > > > good!
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > I will leave my comments in the AIP as well, but
> > at a
> > > > > high
> > > > > > >> level
> > > > > > >> > > >> they are
> > > > > > >> > > >> > > > both relatively focused on the "how", rather than
> > the
> > > > > > "what".
> > > > > > >> > > >> > > > With respect to the pull / polling approach, I
> > > > completely
> > > > > > >> agree
> > > > > > >> > > >> that some
> > > > > > >> > > >> > > > incarnation of this is needed.
> > > > > > >> > > >> > > > I am less certain as to how on this part. The
> > bespoke
> > > > > > >> triggerer
> > > > > > >> > > >> approach
> > > > > > >> > > >> > > > completely makes sense for the long tail here, but
> > > can
> > > > we
> > > > > > do
> > > > > > >> > > better
> > > > > > >> > > >> for the
> > > > > > >> > > >> > > > 20% of scenarios which cover well over 80% of usage
> > > > here
> > > > > is
> > > > > > >> the
> > > > > > >> > > >> question in
> > > > > > >> > > >> > > > my mind. Or, are you thinking of those as being
> > > covered
> > > > > in
> > > > > > >> the
> > > > > > >> > > >> "push"
> > > > > > >> > > >> > > > model?
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Which leads to the "push" model approach.
> > > > > > >> > > >> > > > I am struggling with the same question that Jarek
> > > > raised
> > > > > > here
> > > > > > >> > > about
> > > > > > >> > > >> whether
> > > > > > >> > > >> > > > we need a new Airflow entity over and beyond the
> > > > existing
> > > > > > >> REST API
> > > > > > >> > > >> for the
> > > > > > >> > > >> > > > same.
> > > > > > >> > > >> > > > I am concerned about this becoming a vector of
> > attack
> > > > on
> > > > > > >> Airflow.
> > > > > > >> > > >> > > > I see that this is a hot topic of discussion in the
> > > > > > >> Confluence doc
> > > > > > >> > > >> as well,
> > > > > > >> > > >> > > > but wanted to summarize here as well, so it didn't
> > > get
> > > > > lost
> > > > > > >> in the
> > > > > > >> > > >> threads
> > > > > > >> > > >> > > > of comments.
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > Best regards,
> > > > > > >> > > >> > > > Vikram
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > > > > > >> jarek@potiuk.com>
> > > > > > >> > > >> wrote:
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > > > > Thanks Vincent. I took a look and I have a
> > general
> > > > > > >> comment. I
> > > > > > >> > > >> > > > > strongly think external driven scheduling is
> > really
> > > > > > needed
> > > > > > >> -
> > > > > > >> > > >> especially,
> > > > > > >> > > >> > > > it
> > > > > > >> > > >> > > > > should be much easier for a user to "plug-in"
> > such
> > > an
> > > > > > >> external
> > > > > > >> > > >> event to
> > > > > > >> > > >> > > > > Airflow. And there are two parts of it - as
> > > correctly
> > > > > > >> stated
> > > > > > >> > > >> there - pull
> > > > > > >> > > >> > > > > and push.
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > For the pull - I think it would be great to have
> > a
> > > > kind
> > > > > > of
> > > > > > >> > > >> specialized
> > > > > > >> > > >> > > > > Triggers that will be started when DAG is parsed
> > -
> > > > and
> > > > > > >> those
> > > > > > >> > > >> Triggers
> > > > > > >> > > >> > > > could
> > > > > > >> > > >> > > > > generate the events for DAGs. I think basically
> > > > that's
> > > > > > all
> > > > > > >> that
> > > > > > >> > > is
> > > > > > >> > > >> > > > needed,
> > > > > > >> > > >> > > > > for example I imagine a pubsub trigger that will
> > > > > > subscribe
> > > > > > >> to
> > > > > > >> > > >> messages
> > > > > > >> > > >> > > > > coming on the pubsub queue and fire "Asset" event
> > > > when
> > > > > a
> > > > > > >> message
> > > > > > >> > > >> is
> > > > > > >> > > >> > > > > received. Not much controversy there - I am not
> > > sure
> > > > > > about
> > > > > > >> the
> > > > > > >> > > >> polling
> > > > > > >> > > >> > > > > thing , because I've always believed that when
> > > > > > >> "asyncio-native"
> > > > > > >> > > >> Trigger
> > > > > > >> > > >> > > > is
> > > > > > >> > > >> > > > > run in the asyncio event loop, we do not "poll"
> > > every
> > > > > > >> second or
> > > > > > >> > > >> so (but
> > > > > > >> > > >> > > > > maybe this is just coming from some specific
> > > triggers
> > > > > > that
> > > > > > >> > > >> actually do
> > > > > > >> > > >> > > > > such regular poll. But yes - there are polls
> > like
> > > > > > running
> > > > > > >> > > select
> > > > > > >> > > >> on the
> > > > > > >> > > >> > > > DB
> > > > > > >> > > >> > > > > that cannot be easily "async-ed" so having a
> > > > > configurable
> > > > > > >> > > polling
> > > > > > >> > > >> time
> > > > > > >> > > >> > > > > would be good there (but I am not sure maybe it's
> > > > even
> > > > > > >> possible
> > > > > > >> > > >> today). I
> > > > > > >> > > >> > > > > think this would be really great if we have that
> > > > > option,
> > > > > > >> because
> > > > > > >> > > >> it makes
> > > > > > >> > > >> > > > > it much easier to set up the authorization for
> > > > Airlfow
> > > > > > >> users -
> > > > > > >> > > >> rather
> > > > > > >> > > >> > > > than
> > > > > > >> > > >> > > > > setting up authorization and REST calls coming
> > from
> > > > an
> > > > > > >> external
> > > > > > >> > > >> system,
> > > > > > >> > > >> > > > we
> > > > > > >> > > >> > > > > can utilize Connections of Airlfow to authorize
> > > such
> > > > a
> > > > > > >> Trigger
> > > > > > >> > > to
> > > > > > >> > > >> > > > subscribe
> > > > > > >> > > >> > > > > to events.
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > For the push proposal - as I read the proposal,
> > > the
> > > > > main
> > > > > > >> point
> > > > > > >> > > >> behind it
> > > > > > >> > > >> > > > > is rather than users having to write "Airflow"
> > way
> > > of
> > > > > > >> triggering
> > > > > > >> > > >> events
> > > > > > >> > > >> > > > and
> > > > > > >> > > >> > > > > configuring authentication (using REST API) to
> > > > generate
> > > > > > >> asset
> > > > > > >> > > >> events, is
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > > > make Airflow natively understand external ways of
> > > > > pushing
> > > > > > >> - and
> > > > > > >> > > >> > > > effectively
> > > > > > >> > > >> > > > > authorizing and mapping such incoming
> > unauthorized
> > > > > > >> requests into
> > > > > > >> > > >> event
> > > > > > >> > > >> > > > that
> > > > > > >> > > >> > > > > could be generated by an API REST call.
> > > > > > >> > > >> > > > > I am not really sure honestly if this is
> > something
> > > > that
> > > > > > we
> > > > > > >> want
> > > > > > >> > > as
> > > > > > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such
> > > an
> > > > > > >> > > unauthorised
> > > > > > >> > > >> > > > endpoint
> > > > > > >> > > >> > > > > is probably not a good idea - for a variety of
> > > > reasons,
> > > > > > >> mostly
> > > > > > >> > > >> security.
> > > > > > >> > > >> > > > > And as I understand the goal is that users can
> > > easily
> > > > > > >> point at
> > > > > > >> > > >> > > > "3rd-party"
> > > > > > >> > > >> > > > > notification to Airflow and get the event
> > > generated.
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > My feeling is that while this is needed - it
> > should
> > > > be
> > > > > > >> > > >> externalised from
> > > > > > >> > > >> > > > > airlfow webserver. The authorization has to be
> > set
> > > up
> > > > > > >> anyway
> > > > > > >> > > >> > > > additionally -
> > > > > > >> > > >> > > > > unlike in "poll" case - we cannot use Connections
> > > for
> > > > > > >> > > authorizing
> > > > > > >> > > >> > > > (because
> > > > > > >> > > >> > > > > it's not Airlfow that authorizes in an external
> > > > system
> > > > > -
> > > > > > >> it's
> > > > > > >> > > the
> > > > > > >> > > >> other
> > > > > > >> > > >> > > > way
> > > > > > >> > > >> > > > > round). So we have to anyhow setup "something
> > > extra"
> > > > in
> > > > > > >> Airflow
> > > > > > >> > > to
> > > > > > >> > > >> > > > > authorize the external system. Which could be
> > what
> > > we
> > > > > > have
> > > > > > >> now -
> > > > > > >> > > >> user
> > > > > > >> > > >> > > > that
> > > > > > >> > > >> > > > > allows us to trigger the event. Which means that
> > > our
> > > > > REST
> > > > > > >> API
> > > > > > >> > > >> could
> > > > > > >> > > >> > > > > potentially be used the same way it is now, but
> > we
> > > > will
> > > > > > >> need
> > > > > > >> > > >> "something"
> > > > > > >> > > >> > > > > (library, lambda function etc.) that users could
> > > > easily
> > > > > > >> setup in
> > > > > > >> > > >> the
> > > > > > >> > > >> > > > > external system to map whatever trigger they
> > > generate
> > > > > > >> natively
> > > > > > >> > > >> (say S3
> > > > > > >> > > >> > > > file
> > > > > > >> > > >> > > > > created) to Airflow REST API.
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > As I see it - this is quite often used (and very
> > > > > > >> practical, that
> > > > > > >> > > >> you
> > > > > > >> > > >> > > > deploy
> > > > > > >> > > >> > > > > a cloud function or lambda that subscribes on the
> > > > event
> > > > > > >> received
> > > > > > >> > > >> when
> > > > > > >> > > >> > > > > S3/GCS is created. So it would be on the user to
> > > > deploy
> > > > > > >> such a
> > > > > > >> > > >> lambda -
> > > > > > >> > > >> > > > but
> > > > > > >> > > >> > > > > we **could** provide a library of those: say s3
> > > > lambda,
> > > > > > gcp
> > > > > > >> > > cloud
> > > > > > >> > > >> > > > function
> > > > > > >> > > >> > > > > in respective providers - with documentation how
> > to
> > > > set
> > > > > > >> them up,
> > > > > > >> > > >> and how
> > > > > > >> > > >> > > > to
> > > > > > >> > > >> > > > > configure authorization and we would be generally
> > > > > "done".
> > > > > > >> I am
> > > > > > >> > > >> just not
> > > > > > >> > > >> > > > > sure if we need a new entity in Airflow for that
> > > > (Event
> > > > > > >> > > >> receiver). It
> > > > > > >> > > >> > > > feels
> > > > > > >> > > >> > > > > like it asks Airflow to take more responsibility,
> > > > when
> > > > > we
> > > > > > >> all
> > > > > > >> > > >> think on
> > > > > > >> > > >> > > > what
> > > > > > >> > > >> > > > > to "remove" from Airflow rather than "add" to it
> > -
> > > > > > >> especially
> > > > > > >> > > >> when it
> > > > > > >> > > >> > > > comes
> > > > > > >> > > >> > > > > to external integrations. It feels to me that
> > > Airflow
> > > > > > >> should
> > > > > > >> > > make
> > > > > > >> > > >> it easy
> > > > > > >> > > >> > > > > to be triggered by such an external system and
> > make
> > > > it
> > > > > > >> easy to
> > > > > > >> > > >> "map" to
> > > > > > >> > > >> > > > the
> > > > > > >> > > >> > > > > way we expect to get events triggered, but this
> > > > should
> > > > > be
> > > > > > >> done
> > > > > > >> > > >> outside of
> > > > > > >> > > >> > > > > Airflow. If the users can easily find in our docs
> > > > when
> > > > > > they
> > > > > > >> > > >> search "what
> > > > > > >> > > >> > > > do
> > > > > > >> > > >> > > > > I do to externally trigger Airflow on S3 change":
> > > > > either
> > > > > > a)
> > > > > > >> > > >> configure
> > > > > > >> > > >> > > > > polling in airflow using s3 Connection, or b)
> > > > "create a
> > > > > > >> user +
> > > > > > >> > > >> deploy
> > > > > > >> > > >> > > > this
> > > > > > >> > > >> > > > > lambda with those parameters" - that is "easy
> > > > enough"
> > > > > > and
> > > > > > >> very
> > > > > > >> > > >> practical
> > > > > > >> > > >> > > > > as well.
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > But maybe I am not seeing the whole picture and
> > the
> > > > > real
> > > > > > >> problem
> > > > > > >> > > >> it's
> > > > > > >> > > >> > > > > solving - so take it as a "first review pass" and
> > > > "guts
> > > > > > >> > > feeling".
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > J.
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > > > > >> > > >> > > > <vincbeck@amazon.com.invalid
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > > wrote:
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > > Hello everyone,
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > > > I created a draft AIP regarding "External event
> > > > > driven
> > > > > > >> > > >> scheduling in
> > > > > > >> > > >> > > > > > Airflow". This proposal is about adding
> > > capability
> > > > in
> > > > > > >> Airflow
> > > > > > >> > > to
> > > > > > >> > > >> > > > schedule
> > > > > > >> > > >> > > > > > DAGs based on external events. Here are some
> > > > examples
> > > > > > of
> > > > > > >> such
> > > > > > >> > > >> external
> > > > > > >> > > >> > > > > > events:
> > > > > > >> > > >> > > > > > - A user signs up to one of the user pool
> > defined
> > > > in
> > > > > my
> > > > > > >> cloud
> > > > > > >> > > >> provider
> > > > > > >> > > >> > > > > > - One of the databases used in my company has
> > > been
> > > > > > >> updated
> > > > > > >> > > >> > > > > > - A job in my cloud provider has been executed
> > > > > > >> successfully
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > > > The intent of this AIP is to leverage datasets
> > > > (which
> > > > > > >> will be
> > > > > > >> > > >> soon
> > > > > > >> > > >> > > > > assets)
> > > > > > >> > > >> > > > > > and update them based on external events. I
> > would
> > > > > like
> > > > > > to
> > > > > > >> > > >> propose this
> > > > > > >> > > >> > > > > AIP
> > > > > > >> > > >> > > > > > for discussion and more importantly, hear some
> > > > > > feedbacks
> > > > > > >> from
> > > > > > >> > > >> you :)
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > > > > >> > > >> > > > <
> > > > > > >> > > >> > > >
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > > > Vincent
> > > > > > >> > > >> > > > > >
> > > > > > >> > > >> > > > >
> > > > > > >> > > >> > > >
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >> >
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > >> > To unsubscribe, e-mail:
> > > dev-unsubscribe@airflow.apache.org
> > > > > > >> > > >> > For additional commands, e-mail:
> > > > dev-help@airflow.apache.org
> > > > > > >> > > >> >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > > >>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> > > >> To unsubscribe, e-mail:
> > dev-unsubscribe@airflow.apache.org
> > > > > > >> > > >> For additional commands, e-mail:
> > > dev-help@airflow.apache.org
> > > > > > >> > > >>
> > > > > > >> > > >>
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > ---------------------------------------------------------------------
> > > > > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Vincent and Kaxil, Agree it can be added to future work.
Looking at discussion on event mapping, Today we know the source systems
and we are aware of the schemas from source systems (S3, SQS, Event Arc
etc;)
However, as new services emerge and their schemas evolve, this may change.
Agree with Kaxil has mentioned that transformers are very effective in
converting the required format before sending data to target systems.
Is it a good idea to define a schema for events to enforce from airflow?
There are some open standards, such as CloudEvents[1], that we could
consider.
This approach would allow us to maintain versions (v1, v2, ... vn) as we
evolve, providing utilities that users can easily plug into and send events
to Airflow.
It would also simplify maintenance within the Airflow system to handle
incoming events because we get a known version of the schema from
user request.
If this is already being addressed, please disregard my comments.
[1]: https://github.com/cloudevents/spec
Regards,
Pavan
On Fri, Aug 2, 2024 at 1:40 AM Kaxil Naik <ka...@gmail.com> wrote:
> Yes but the big difference is you will create a single user for EventBridge
> since that is the one sending request to Airflow, single user for EventArc
> for GCP and 1 user for every other EventListener or application ---- as
> compared to 1 user per type of Payload (since Airflow will need to
> understand the payload of the original source). So in that case, you will
> have 1 user,function mapping for S3, 1user,function mapping for Redshift,
> and it goes on. The former approach is also consistent with our Connection
> model, where we have one standardized AWS & GCP connection that works for
> most, if not all, services.
>
> but the user will have to be added anyway (some kind of
> > service account - because the API needs to be authorized - that part is
> not
> > changed). (unless of course you want to use the same user for all kinds
> of
> > external interfaces, which for security point of view is a very bad idea
> -
> > each external system should have their own "service account" - that 's
> the
> > best practices from the security point of view.
>
>
> On Fri, 2 Aug 2024 at 01:22, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I am all for it - if we want to stick to event bridge or similar and
> > recommend it to our users, it's perfectly fine for me, It would be great
> > however to add a documentation explaining the steps and some examples -
> > ideally for most of our providers and "standard" ways of triggering such
> an
> > event. This is even what I proposed originally whent the first version of
> > the document was created ( to just document how to map the events
> > externally).
> >
> > BTW. Yes - in this case you need to implement the logic in the
> > event bridge, but the user will have to be added anyway (some kind of
> > service account - because the API needs to be authorized - that part is
> not
> > changed). (unless of course you want to use the same user for all kinds
> of
> > external interfaces, which for security point of view is a very bad idea
> -
> > each external system should have their own "service account" - that 's
> the
> > best practices from the security point of view.
> >
> > J,
> >
> > On Fri, Aug 2, 2024 at 2:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > I was discussing this with Vincent. In either case, same as now or the
> > one
> > > proposed in the AIP, a user will have to use something like AWS
> > > EventBridge[1] or GCS EventArc [2] where users will consume the event
> > from
> > > object storage (S3 object creation for example), and then they will
> have
> > to
> > > add the Airflow's Create Dataset endpoint to EventBridge [3]. Now, if
> you
> > > just customize the payload to build the URI which is allowed (either
> via
> > > GET / POST) in eventbridge, it works right now. However, with the
> current
> > > proposal: a user will have to create a new user in Airflow and some
> > mapping
> > > to a function (that is either in the provider or a new user-defined
> > > function) that can understand this specific payload, in this example
> the
> > > payload for S3 events. This will become huge because this means that
> for
> > > each payload, we will have to provide a new function and keep it
> updated.
> > > From users POV, they will need to create a new user every time for a
> new
> > > service (S3, Redshift, SNS, Bedrock etc). This will again likely have
> to
> > go
> > > to the Auth manager backend. Compared to what's available today -- i.e.
> > > building a URI & extra metadata that can not only work with EventBridge
> > or
> > > Eventarc but by any service.
> > >
> > > Since we already have to use things like EventBridge or EventArc for
> > > managed service providers to transform the event, it fits well with the
> > > existing approach. AWS Blog [3] even has a similar example for Datadog
> > > where they use input transformer "{"detail":"$.detail"}" before sending
> > it
> > > to Datadog's API.
> > > >"Having producer of an event generating even in their standard way,
> make
> > > it easy for airflow to consume it as a dataset event without external
> > > entities":
> > >
> > >
> > > [1]: https://aws.amazon.com/eventbridge/ |
> > > https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
> > > [2]: https://cloud.google.com/eventarc/docs
> > > [3]:
> > >
> > >
> >
> https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
> > >
> > > On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > I proposed the mapping - because it's the easiest way (I think) to
> map
> > > > between the "native" source to "airflow" target expectations. There
> are
> > > > many producers of such events, and Airflow is the consumer. And it
> > seems
> > > > appropriate to have a way for our users to easily plug events
> produced
> > > from
> > > > one system into our "events" API - without having to employ external
> > > > "mapper" (say lambda) doing the conversion. While I think it is
> indeed
> > "a
> > > > bit odd", it's a solution that might leverage most of what we have -
> > > > authorisation and API exposure via "user" in API.
> > > >
> > > > While I - myself - find it it a bit unusual, I think it might do the
> > job,
> > > > But I wonder if there is any alternative solution to the problem of
> > > "Having
> > > > producer of an event generating even in their standard way, make it
> > easy
> > > > for airflow to consume it as a dataset event without external
> > entities".
> > > >
> > > > On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >
> > > > > I would love for VOTE to get started on this one. I think most of
> the
> > > > > commenters and those who replied to this email are happy with the
> > > > proposal
> > > > > on the poll-based approach.
> > > > >
> > > > > Regarding the push-based approach, I am not convinced that the
> > proposed
> > > > > implementation has any gains over what's already available with the
> > > > Dataset
> > > > > Event Create API; the one user-to-one function mapping is an odd
> user
> > > > > experience. I'm curious to hear what others think.
> > > > >
> > > > > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com>
> wrote:
> > > > >
> > > > > > I agree with both of you that it is indeed a good idea and that
> it
> > > can
> > > > be
> > > > > > added in Future work -- doesn't need to be part of this AIP.
> > > > > >
> > > > > > Thanks for the interest. I was not aware of such feature and this
> > > looks
> > > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > > especially
> > > > > >> for testing when you can easily replay events received in the
> > past.
> > > > > >> However, I do not think it should be part of the AIP and, as you
> > > > > mentioned,
> > > > > >> if should be a future work or a follow-up item of the AIP.
> Please
> > > let
> > > > me
> > > > > >> know if you (or anyone) disagree with this and we can talk about
> > it.
> > > > > >> Otherwise I'll update the future work section of the AIP and
> > mention
> > > > > this
> > > > > >> archive and replay feature.
> > > > > >
> > > > > >
> > > > > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org>
> > > wrote:
> > > > > >
> > > > > >> Hey Pavan,
> > > > > >>
> > > > > >> Thanks for the interest. I was not aware of such feature and
> this
> > > > looks
> > > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > > especially
> > > > > >> for testing when you can easily replay events received in the
> > past.
> > > > > >> However, I do not think it should be part of the AIP and, as you
> > > > > mentioned,
> > > > > >> if should be a future work or a follow-up item of the AIP.
> Please
> > > let
> > > > me
> > > > > >> know if you (or anyone) disagree with this and we can talk about
> > it.
> > > > > >> Otherwise I'll update the future work section of the AIP and
> > mention
> > > > > this
> > > > > >> archive and replay feature.
> > > > > >>
> > > > > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > > > > >> > Thanks Vincent, I took a look , this is really good. Don't
> have
> > > > access
> > > > > >> to
> > > > > >> > the confluence page to comment :) so adding it here.
> > > > > >> >
> > > > > >> > As events arrive-->do somework-->end.
> > > > > >> >
> > > > > >> > So I'm uncertain if my comment pertains to the current
> poll/push
> > > > model
> > > > > >> or
> > > > > >> > if it fits part of future work(seen event batching ).
> > > > > >> >
> > > > > >> > Have you given any thought to the event archival mechanism and
> > > event
> > > > > >> > replay? This could significantly aid in testing and recovery
> of
> > > > > workflow
> > > > > >> > and testing new functionality with events by just replay the
> > > events.
> > > > > The
> > > > > >> > archival mechanism I am referring to is similar to today in
> AWS
> > we
> > > > > have
> > > > > >> > Event Bridge Archive and Replay.
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> > Pavan
> > > > > >> >
> > > > > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <
> kaxilnaik@gmail.com>
> > > > > wrote:
> > > > > >> >
> > > > > >> > > I actually did manage to take a look, thanks for the work. I
> > am
> > > +1
> > > > > on
> > > > > >> the
> > > > > >> > > poll-based approach -- left a comment on the push-based: I
> am
> > > not
> > > > > >> sure of
> > > > > >> > > why we need a function since create asset event API endpoint
> > > > should
> > > > > >> have
> > > > > >> > > all info needed for what the Asset was.
> > > > > >> > >
> > > > > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <
> kaxilnaik@gmail.com>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > > > > >> > > >
> > > > > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <
> > > vincbeck@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >> > > >
> > > > > >> > > >> Hi everyone,
> > > > > >> > > >>
> > > > > >> > > >> I updated the AIP-82 given the different comments and
> > > concerns
> > > > I
> > > > > >> > > >> received. I also tried to reply to all comments
> > > individually. I
> > > > > >> would
> > > > > >> > > >> really appreciate if you can do a second pass and let me
> > know
> > > > > what
> > > > > >> you
> > > > > >> > > >> think. Overall, this is what I changed in the AIP:
> > > > > >> > > >>
> > > > > >> > > >> - Push based event-driven scheduling. I updated this
> > section
> > > > > >> entirely
> > > > > >> > > >> because I received many concerns about the previous
> > proposal.
> > > > The
> > > > > >> > > overall
> > > > > >> > > >> idea now is to leverage the create asset event API
> endpoint
> > > to
> > > > > send
> > > > > >> > > >> notifications from external (e.g. cloud provider) to
> > Airflow
> > > > > >> > > environment.
> > > > > >> > > >>
> > > > > >> > > >> - I updated the poll based event-driven scheduling DAG
> > author
> > > > > >> experience
> > > > > >> > > >> to use a message queue scenario. I understood that this
> is
> > > > > >> probably the
> > > > > >> > > >> main use case we are trying to cover with this AIP, thus
> I
> > > used
> > > > > it
> > > > > >> as
> > > > > >> > > >> example and mentioned it multiple times across the AIP.
> > > > > >> > > >>
> > > > > >> > > >> Thanks again for your time :)
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > > >> > > >>
> > > > > >> > > >> Vincent
> > > > > >> > > >>
> > > > > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > > > >> > > >> > Thanks a lot all for the comments, this is very much
> > > > > >> appreciated! I
> > > > > >> > > >> received many comments from this thread and in
> confluence,
> > > > thanks
> > > > > >> again.
> > > > > >> > > >> I'll try to address them all in the AIP and will send an
> > > email
> > > > in
> > > > > >> this
> > > > > >> > > >> thread once done. I will most likely revisit the
> push-based
> > > > > >> approach
> > > > > >> > > given
> > > > > >> > > >> the number of concerns I received, thanks Jarek for
> > proposing
> > > > > >> another
> > > > > >> > > >> solution, I'll probably go down that path.
> > > > > >> > > >> >
> > > > > >> > > >> > One follow-up question Vikram.
> > > > > >> > > >> >
> > > > > >> > > >> > > The bespoke triggerer approach completely makes sense
> > for
> > > > the
> > > > > >> long
> > > > > >> > > >> tail here, but can we do better for the 20% of scenarios
> > > which
> > > > > >> cover
> > > > > >> > > well
> > > > > >> > > >> over 80% of usage here is the question in my mind. Or,
> are
> > > you
> > > > > >> thinking
> > > > > >> > > of
> > > > > >> > > >> those as being covered in the "push" model?
> > > > > >> > > >> >
> > > > > >> > > >> > Could you share more details about what is this "20% of
> > > > > scenarios
> > > > > >> > > which
> > > > > >> > > >> cover well over 80% of usage" please?
> > > > > >> > > >> >
> > > > > >> > > >> > Vincent
> > > > > >> > > >> >
> > > > > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > > > >> > > >> > > Thanks Vincent for driving these, I have added my
> > > comments
> > > > to
> > > > > >> the
> > > > > >> > > AIP
> > > > > >> > > >> too.
> > > > > >> > > >> > >
> > > > > >> > > >> > > Regards,
> > > > > >> > > >> > > Kaxil
> > > > > >> > > >> > >
> > > > > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > > > > (XC-AS/EAE-ADA-T)
> > > > > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > > > >> > > >> > >
> > > > > >> > > >> > > > +1 on the comments of Vikram and Jarek, added main
> > > points
> > > > > on
> > > > > >> > > >> confluence
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > > > >> > > >> > > > ________________________________
> > > > > >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > > > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > > > >> > > >> > > > To: dev@airflow.apache.org <dev@airflow.apache.org
> >
> > > > > >> > > >> > > > Subject: Re: [DISCUSS] External event driven
> > scheduling
> > > > in
> > > > > >> Airflow
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Vincent,
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Thanks for writing this up. The overview looks
> really
> > > > good!
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > I will leave my comments in the AIP as well, but
> at a
> > > > high
> > > > > >> level
> > > > > >> > > >> they are
> > > > > >> > > >> > > > both relatively focused on the "how", rather than
> the
> > > > > "what".
> > > > > >> > > >> > > > With respect to the pull / polling approach, I
> > > completely
> > > > > >> agree
> > > > > >> > > >> that some
> > > > > >> > > >> > > > incarnation of this is needed.
> > > > > >> > > >> > > > I am less certain as to how on this part. The
> bespoke
> > > > > >> triggerer
> > > > > >> > > >> approach
> > > > > >> > > >> > > > completely makes sense for the long tail here, but
> > can
> > > we
> > > > > do
> > > > > >> > > better
> > > > > >> > > >> for the
> > > > > >> > > >> > > > 20% of scenarios which cover well over 80% of usage
> > > here
> > > > is
> > > > > >> the
> > > > > >> > > >> question in
> > > > > >> > > >> > > > my mind. Or, are you thinking of those as being
> > covered
> > > > in
> > > > > >> the
> > > > > >> > > >> "push"
> > > > > >> > > >> > > > model?
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Which leads to the "push" model approach.
> > > > > >> > > >> > > > I am struggling with the same question that Jarek
> > > raised
> > > > > here
> > > > > >> > > about
> > > > > >> > > >> whether
> > > > > >> > > >> > > > we need a new Airflow entity over and beyond the
> > > existing
> > > > > >> REST API
> > > > > >> > > >> for the
> > > > > >> > > >> > > > same.
> > > > > >> > > >> > > > I am concerned about this becoming a vector of
> attack
> > > on
> > > > > >> Airflow.
> > > > > >> > > >> > > > I see that this is a hot topic of discussion in the
> > > > > >> Confluence doc
> > > > > >> > > >> as well,
> > > > > >> > > >> > > > but wanted to summarize here as well, so it didn't
> > get
> > > > lost
> > > > > >> in the
> > > > > >> > > >> threads
> > > > > >> > > >> > > > of comments.
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > Best regards,
> > > > > >> > > >> > > > Vikram
> > > > > >> > > >> > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > > > > >> jarek@potiuk.com>
> > > > > >> > > >> wrote:
> > > > > >> > > >> > > >
> > > > > >> > > >> > > > > Thanks Vincent. I took a look and I have a
> general
> > > > > >> comment. I
> > > > > >> > > >> > > > > strongly think external driven scheduling is
> really
> > > > > needed
> > > > > >> -
> > > > > >> > > >> especially,
> > > > > >> > > >> > > > it
> > > > > >> > > >> > > > > should be much easier for a user to "plug-in"
> such
> > an
> > > > > >> external
> > > > > >> > > >> event to
> > > > > >> > > >> > > > > Airflow. And there are two parts of it - as
> > correctly
> > > > > >> stated
> > > > > >> > > >> there - pull
> > > > > >> > > >> > > > > and push.
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > For the pull - I think it would be great to have
> a
> > > kind
> > > > > of
> > > > > >> > > >> specialized
> > > > > >> > > >> > > > > Triggers that will be started when DAG is parsed
> -
> > > and
> > > > > >> those
> > > > > >> > > >> Triggers
> > > > > >> > > >> > > > could
> > > > > >> > > >> > > > > generate the events for DAGs. I think basically
> > > that's
> > > > > all
> > > > > >> that
> > > > > >> > > is
> > > > > >> > > >> > > > needed,
> > > > > >> > > >> > > > > for example I imagine a pubsub trigger that will
> > > > > subscribe
> > > > > >> to
> > > > > >> > > >> messages
> > > > > >> > > >> > > > > coming on the pubsub queue and fire "Asset" event
> > > when
> > > > a
> > > > > >> message
> > > > > >> > > >> is
> > > > > >> > > >> > > > > received. Not much controversy there - I am not
> > sure
> > > > > about
> > > > > >> the
> > > > > >> > > >> polling
> > > > > >> > > >> > > > > thing , because I've always believed that when
> > > > > >> "asyncio-native"
> > > > > >> > > >> Trigger
> > > > > >> > > >> > > > is
> > > > > >> > > >> > > > > run in the asyncio event loop, we do not "poll"
> > every
> > > > > >> second or
> > > > > >> > > >> so (but
> > > > > >> > > >> > > > > maybe this is just coming from some specific
> > triggers
> > > > > that
> > > > > >> > > >> actually do
> > > > > >> > > >> > > > > such regular poll. But yes - there are polls
> like
> > > > > running
> > > > > >> > > select
> > > > > >> > > >> on the
> > > > > >> > > >> > > > DB
> > > > > >> > > >> > > > > that cannot be easily "async-ed" so having a
> > > > configurable
> > > > > >> > > polling
> > > > > >> > > >> time
> > > > > >> > > >> > > > > would be good there (but I am not sure maybe it's
> > > even
> > > > > >> possible
> > > > > >> > > >> today). I
> > > > > >> > > >> > > > > think this would be really great if we have that
> > > > option,
> > > > > >> because
> > > > > >> > > >> it makes
> > > > > >> > > >> > > > > it much easier to set up the authorization for
> > > Airlfow
> > > > > >> users -
> > > > > >> > > >> rather
> > > > > >> > > >> > > > than
> > > > > >> > > >> > > > > setting up authorization and REST calls coming
> from
> > > an
> > > > > >> external
> > > > > >> > > >> system,
> > > > > >> > > >> > > > we
> > > > > >> > > >> > > > > can utilize Connections of Airlfow to authorize
> > such
> > > a
> > > > > >> Trigger
> > > > > >> > > to
> > > > > >> > > >> > > > subscribe
> > > > > >> > > >> > > > > to events.
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > For the push proposal - as I read the proposal,
> > the
> > > > main
> > > > > >> point
> > > > > >> > > >> behind it
> > > > > >> > > >> > > > > is rather than users having to write "Airflow"
> way
> > of
> > > > > >> triggering
> > > > > >> > > >> events
> > > > > >> > > >> > > > and
> > > > > >> > > >> > > > > configuring authentication (using REST API) to
> > > generate
> > > > > >> asset
> > > > > >> > > >> events, is
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > > > make Airflow natively understand external ways of
> > > > pushing
> > > > > >> - and
> > > > > >> > > >> > > > effectively
> > > > > >> > > >> > > > > authorizing and mapping such incoming
> unauthorized
> > > > > >> requests into
> > > > > >> > > >> event
> > > > > >> > > >> > > > that
> > > > > >> > > >> > > > > could be generated by an API REST call.
> > > > > >> > > >> > > > > I am not really sure honestly if this is
> something
> > > that
> > > > > we
> > > > > >> want
> > > > > >> > > as
> > > > > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such
> > an
> > > > > >> > > unauthorised
> > > > > >> > > >> > > > endpoint
> > > > > >> > > >> > > > > is probably not a good idea - for a variety of
> > > reasons,
> > > > > >> mostly
> > > > > >> > > >> security.
> > > > > >> > > >> > > > > And as I understand the goal is that users can
> > easily
> > > > > >> point at
> > > > > >> > > >> > > > "3rd-party"
> > > > > >> > > >> > > > > notification to Airflow and get the event
> > generated.
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > My feeling is that while this is needed - it
> should
> > > be
> > > > > >> > > >> externalised from
> > > > > >> > > >> > > > > airlfow webserver. The authorization has to be
> set
> > up
> > > > > >> anyway
> > > > > >> > > >> > > > additionally -
> > > > > >> > > >> > > > > unlike in "poll" case - we cannot use Connections
> > for
> > > > > >> > > authorizing
> > > > > >> > > >> > > > (because
> > > > > >> > > >> > > > > it's not Airlfow that authorizes in an external
> > > system
> > > > -
> > > > > >> it's
> > > > > >> > > the
> > > > > >> > > >> other
> > > > > >> > > >> > > > way
> > > > > >> > > >> > > > > round). So we have to anyhow setup "something
> > extra"
> > > in
> > > > > >> Airflow
> > > > > >> > > to
> > > > > >> > > >> > > > > authorize the external system. Which could be
> what
> > we
> > > > > have
> > > > > >> now -
> > > > > >> > > >> user
> > > > > >> > > >> > > > that
> > > > > >> > > >> > > > > allows us to trigger the event. Which means that
> > our
> > > > REST
> > > > > >> API
> > > > > >> > > >> could
> > > > > >> > > >> > > > > potentially be used the same way it is now, but
> we
> > > will
> > > > > >> need
> > > > > >> > > >> "something"
> > > > > >> > > >> > > > > (library, lambda function etc.) that users could
> > > easily
> > > > > >> setup in
> > > > > >> > > >> the
> > > > > >> > > >> > > > > external system to map whatever trigger they
> > generate
> > > > > >> natively
> > > > > >> > > >> (say S3
> > > > > >> > > >> > > > file
> > > > > >> > > >> > > > > created) to Airflow REST API.
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > As I see it - this is quite often used (and very
> > > > > >> practical, that
> > > > > >> > > >> you
> > > > > >> > > >> > > > deploy
> > > > > >> > > >> > > > > a cloud function or lambda that subscribes on the
> > > event
> > > > > >> received
> > > > > >> > > >> when
> > > > > >> > > >> > > > > S3/GCS is created. So it would be on the user to
> > > deploy
> > > > > >> such a
> > > > > >> > > >> lambda -
> > > > > >> > > >> > > > but
> > > > > >> > > >> > > > > we **could** provide a library of those: say s3
> > > lambda,
> > > > > gcp
> > > > > >> > > cloud
> > > > > >> > > >> > > > function
> > > > > >> > > >> > > > > in respective providers - with documentation how
> to
> > > set
> > > > > >> them up,
> > > > > >> > > >> and how
> > > > > >> > > >> > > > to
> > > > > >> > > >> > > > > configure authorization and we would be generally
> > > > "done".
> > > > > >> I am
> > > > > >> > > >> just not
> > > > > >> > > >> > > > > sure if we need a new entity in Airflow for that
> > > (Event
> > > > > >> > > >> receiver). It
> > > > > >> > > >> > > > feels
> > > > > >> > > >> > > > > like it asks Airflow to take more responsibility,
> > > when
> > > > we
> > > > > >> all
> > > > > >> > > >> think on
> > > > > >> > > >> > > > what
> > > > > >> > > >> > > > > to "remove" from Airflow rather than "add" to it
> -
> > > > > >> especially
> > > > > >> > > >> when it
> > > > > >> > > >> > > > comes
> > > > > >> > > >> > > > > to external integrations. It feels to me that
> > Airflow
> > > > > >> should
> > > > > >> > > make
> > > > > >> > > >> it easy
> > > > > >> > > >> > > > > to be triggered by such an external system and
> make
> > > it
> > > > > >> easy to
> > > > > >> > > >> "map" to
> > > > > >> > > >> > > > the
> > > > > >> > > >> > > > > way we expect to get events triggered, but this
> > > should
> > > > be
> > > > > >> done
> > > > > >> > > >> outside of
> > > > > >> > > >> > > > > Airflow. If the users can easily find in our docs
> > > when
> > > > > they
> > > > > >> > > >> search "what
> > > > > >> > > >> > > > do
> > > > > >> > > >> > > > > I do to externally trigger Airflow on S3 change":
> > > > either
> > > > > a)
> > > > > >> > > >> configure
> > > > > >> > > >> > > > > polling in airflow using s3 Connection, or b)
> > > "create a
> > > > > >> user +
> > > > > >> > > >> deploy
> > > > > >> > > >> > > > this
> > > > > >> > > >> > > > > lambda with those parameters" - that is "easy
> > > enough"
> > > > > and
> > > > > >> very
> > > > > >> > > >> practical
> > > > > >> > > >> > > > > as well.
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > But maybe I am not seeing the whole picture and
> the
> > > > real
> > > > > >> problem
> > > > > >> > > >> it's
> > > > > >> > > >> > > > > solving - so take it as a "first review pass" and
> > > "guts
> > > > > >> > > feeling".
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > J.
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > > > >> > > >> > > > <vincbeck@amazon.com.invalid
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > > wrote:
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > > Hello everyone,
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > > > I created a draft AIP regarding "External event
> > > > driven
> > > > > >> > > >> scheduling in
> > > > > >> > > >> > > > > > Airflow". This proposal is about adding
> > capability
> > > in
> > > > > >> Airflow
> > > > > >> > > to
> > > > > >> > > >> > > > schedule
> > > > > >> > > >> > > > > > DAGs based on external events. Here are some
> > > examples
> > > > > of
> > > > > >> such
> > > > > >> > > >> external
> > > > > >> > > >> > > > > > events:
> > > > > >> > > >> > > > > > - A user signs up to one of the user pool
> defined
> > > in
> > > > my
> > > > > >> cloud
> > > > > >> > > >> provider
> > > > > >> > > >> > > > > > - One of the databases used in my company has
> > been
> > > > > >> updated
> > > > > >> > > >> > > > > > - A job in my cloud provider has been executed
> > > > > >> successfully
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > > > The intent of this AIP is to leverage datasets
> > > (which
> > > > > >> will be
> > > > > >> > > >> soon
> > > > > >> > > >> > > > > assets)
> > > > > >> > > >> > > > > > and update them based on external events. I
> would
> > > > like
> > > > > to
> > > > > >> > > >> propose this
> > > > > >> > > >> > > > > AIP
> > > > > >> > > >> > > > > > for discussion and more importantly, hear some
> > > > > feedbacks
> > > > > >> from
> > > > > >> > > >> you :)
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > > > >> > > >> > > > <
> > > > > >> > > >> > > >
> > > > > >> > > >>
> > > > > >> > >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > > > Vincent
> > > > > >> > > >> > > > > >
> > > > > >> > > >> > > > >
> > > > > >> > > >> > > >
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > >> > To unsubscribe, e-mail:
> > dev-unsubscribe@airflow.apache.org
> > > > > >> > > >> > For additional commands, e-mail:
> > > dev-help@airflow.apache.org
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> > > >> To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > > >> > > >> For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > > >> > > >>
> > > > > >> > > >>
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
Yes but the big difference is you will create a single user for EventBridge
since that is the one sending request to Airflow, single user for EventArc
for GCP and 1 user for every other EventListener or application ---- as
compared to 1 user per type of Payload (since Airflow will need to
understand the payload of the original source). So in that case, you will
have 1 user,function mapping for S3, 1user,function mapping for Redshift,
and it goes on. The former approach is also consistent with our Connection
model, where we have one standardized AWS & GCP connection that works for
most, if not all, services.
but the user will have to be added anyway (some kind of
> service account - because the API needs to be authorized - that part is not
> changed). (unless of course you want to use the same user for all kinds of
> external interfaces, which for security point of view is a very bad idea -
> each external system should have their own "service account" - that 's the
> best practices from the security point of view.
On Fri, 2 Aug 2024 at 01:22, Jarek Potiuk <ja...@potiuk.com> wrote:
> I am all for it - if we want to stick to event bridge or similar and
> recommend it to our users, it's perfectly fine for me, It would be great
> however to add a documentation explaining the steps and some examples -
> ideally for most of our providers and "standard" ways of triggering such an
> event. This is even what I proposed originally whent the first version of
> the document was created ( to just document how to map the events
> externally).
>
> BTW. Yes - in this case you need to implement the logic in the
> event bridge, but the user will have to be added anyway (some kind of
> service account - because the API needs to be authorized - that part is not
> changed). (unless of course you want to use the same user for all kinds of
> external interfaces, which for security point of view is a very bad idea -
> each external system should have their own "service account" - that 's the
> best practices from the security point of view.
>
> J,
>
> On Fri, Aug 2, 2024 at 2:01 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > I was discussing this with Vincent. In either case, same as now or the
> one
> > proposed in the AIP, a user will have to use something like AWS
> > EventBridge[1] or GCS EventArc [2] where users will consume the event
> from
> > object storage (S3 object creation for example), and then they will have
> to
> > add the Airflow's Create Dataset endpoint to EventBridge [3]. Now, if you
> > just customize the payload to build the URI which is allowed (either via
> > GET / POST) in eventbridge, it works right now. However, with the current
> > proposal: a user will have to create a new user in Airflow and some
> mapping
> > to a function (that is either in the provider or a new user-defined
> > function) that can understand this specific payload, in this example the
> > payload for S3 events. This will become huge because this means that for
> > each payload, we will have to provide a new function and keep it updated.
> > From users POV, they will need to create a new user every time for a new
> > service (S3, Redshift, SNS, Bedrock etc). This will again likely have to
> go
> > to the Auth manager backend. Compared to what's available today -- i.e.
> > building a URI & extra metadata that can not only work with EventBridge
> or
> > Eventarc but by any service.
> >
> > Since we already have to use things like EventBridge or EventArc for
> > managed service providers to transform the event, it fits well with the
> > existing approach. AWS Blog [3] even has a similar example for Datadog
> > where they use input transformer "{"detail":"$.detail"}" before sending
> it
> > to Datadog's API.
> > >"Having producer of an event generating even in their standard way, make
> > it easy for airflow to consume it as a dataset event without external
> > entities":
> >
> >
> > [1]: https://aws.amazon.com/eventbridge/ |
> > https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
> > [2]: https://cloud.google.com/eventarc/docs
> > [3]:
> >
> >
> https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
> >
> > On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I proposed the mapping - because it's the easiest way (I think) to map
> > > between the "native" source to "airflow" target expectations. There are
> > > many producers of such events, and Airflow is the consumer. And it
> seems
> > > appropriate to have a way for our users to easily plug events produced
> > from
> > > one system into our "events" API - without having to employ external
> > > "mapper" (say lambda) doing the conversion. While I think it is indeed
> "a
> > > bit odd", it's a solution that might leverage most of what we have -
> > > authorisation and API exposure via "user" in API.
> > >
> > > While I - myself - find it it a bit unusual, I think it might do the
> job,
> > > But I wonder if there is any alternative solution to the problem of
> > "Having
> > > producer of an event generating even in their standard way, make it
> easy
> > > for airflow to consume it as a dataset event without external
> entities".
> > >
> > > On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > I would love for VOTE to get started on this one. I think most of the
> > > > commenters and those who replied to this email are happy with the
> > > proposal
> > > > on the poll-based approach.
> > > >
> > > > Regarding the push-based approach, I am not convinced that the
> proposed
> > > > implementation has any gains over what's already available with the
> > > Dataset
> > > > Event Create API; the one user-to-one function mapping is an odd user
> > > > experience. I'm curious to hear what others think.
> > > >
> > > > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com> wrote:
> > > >
> > > > > I agree with both of you that it is indeed a good idea and that it
> > can
> > > be
> > > > > added in Future work -- doesn't need to be part of this AIP.
> > > > >
> > > > > Thanks for the interest. I was not aware of such feature and this
> > looks
> > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > especially
> > > > >> for testing when you can easily replay events received in the
> past.
> > > > >> However, I do not think it should be part of the AIP and, as you
> > > > mentioned,
> > > > >> if should be a future work or a follow-up item of the AIP. Please
> > let
> > > me
> > > > >> know if you (or anyone) disagree with this and we can talk about
> it.
> > > > >> Otherwise I'll update the future work section of the AIP and
> mention
> > > > this
> > > > >> archive and replay feature.
> > > > >
> > > > >
> > > > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org>
> > wrote:
> > > > >
> > > > >> Hey Pavan,
> > > > >>
> > > > >> Thanks for the interest. I was not aware of such feature and this
> > > looks
> > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > especially
> > > > >> for testing when you can easily replay events received in the
> past.
> > > > >> However, I do not think it should be part of the AIP and, as you
> > > > mentioned,
> > > > >> if should be a future work or a follow-up item of the AIP. Please
> > let
> > > me
> > > > >> know if you (or anyone) disagree with this and we can talk about
> it.
> > > > >> Otherwise I'll update the future work section of the AIP and
> mention
> > > > this
> > > > >> archive and replay feature.
> > > > >>
> > > > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > > > >> > Thanks Vincent, I took a look , this is really good. Don't have
> > > access
> > > > >> to
> > > > >> > the confluence page to comment :) so adding it here.
> > > > >> >
> > > > >> > As events arrive-->do somework-->end.
> > > > >> >
> > > > >> > So I'm uncertain if my comment pertains to the current poll/push
> > > model
> > > > >> or
> > > > >> > if it fits part of future work(seen event batching ).
> > > > >> >
> > > > >> > Have you given any thought to the event archival mechanism and
> > event
> > > > >> > replay? This could significantly aid in testing and recovery of
> > > > workflow
> > > > >> > and testing new functionality with events by just replay the
> > events.
> > > > The
> > > > >> > archival mechanism I am referring to is similar to today in AWS
> we
> > > > have
> > > > >> > Event Bridge Archive and Replay.
> > > > >> >
> > > > >> > Regards,
> > > > >> > Pavan
> > > > >> >
> > > > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > I actually did manage to take a look, thanks for the work. I
> am
> > +1
> > > > on
> > > > >> the
> > > > >> > > poll-based approach -- left a comment on the push-based: I am
> > not
> > > > >> sure of
> > > > >> > > why we need a function since create asset event API endpoint
> > > should
> > > > >> have
> > > > >> > > all info needed for what the Asset was.
> > > > >> > >
> > > > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > > > >> > > >
> > > > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <
> > vincbeck@apache.org
> > > >
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > >> Hi everyone,
> > > > >> > > >>
> > > > >> > > >> I updated the AIP-82 given the different comments and
> > concerns
> > > I
> > > > >> > > >> received. I also tried to reply to all comments
> > individually. I
> > > > >> would
> > > > >> > > >> really appreciate if you can do a second pass and let me
> know
> > > > what
> > > > >> you
> > > > >> > > >> think. Overall, this is what I changed in the AIP:
> > > > >> > > >>
> > > > >> > > >> - Push based event-driven scheduling. I updated this
> section
> > > > >> entirely
> > > > >> > > >> because I received many concerns about the previous
> proposal.
> > > The
> > > > >> > > overall
> > > > >> > > >> idea now is to leverage the create asset event API endpoint
> > to
> > > > send
> > > > >> > > >> notifications from external (e.g. cloud provider) to
> Airflow
> > > > >> > > environment.
> > > > >> > > >>
> > > > >> > > >> - I updated the poll based event-driven scheduling DAG
> author
> > > > >> experience
> > > > >> > > >> to use a message queue scenario. I understood that this is
> > > > >> probably the
> > > > >> > > >> main use case we are trying to cover with this AIP, thus I
> > used
> > > > it
> > > > >> as
> > > > >> > > >> example and mentioned it multiple times across the AIP.
> > > > >> > > >>
> > > > >> > > >> Thanks again for your time :)
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > >> > > >>
> > > > >> > > >> Vincent
> > > > >> > > >>
> > > > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > > >> > > >> > Thanks a lot all for the comments, this is very much
> > > > >> appreciated! I
> > > > >> > > >> received many comments from this thread and in confluence,
> > > thanks
> > > > >> again.
> > > > >> > > >> I'll try to address them all in the AIP and will send an
> > email
> > > in
> > > > >> this
> > > > >> > > >> thread once done. I will most likely revisit the push-based
> > > > >> approach
> > > > >> > > given
> > > > >> > > >> the number of concerns I received, thanks Jarek for
> proposing
> > > > >> another
> > > > >> > > >> solution, I'll probably go down that path.
> > > > >> > > >> >
> > > > >> > > >> > One follow-up question Vikram.
> > > > >> > > >> >
> > > > >> > > >> > > The bespoke triggerer approach completely makes sense
> for
> > > the
> > > > >> long
> > > > >> > > >> tail here, but can we do better for the 20% of scenarios
> > which
> > > > >> cover
> > > > >> > > well
> > > > >> > > >> over 80% of usage here is the question in my mind. Or, are
> > you
> > > > >> thinking
> > > > >> > > of
> > > > >> > > >> those as being covered in the "push" model?
> > > > >> > > >> >
> > > > >> > > >> > Could you share more details about what is this "20% of
> > > > scenarios
> > > > >> > > which
> > > > >> > > >> cover well over 80% of usage" please?
> > > > >> > > >> >
> > > > >> > > >> > Vincent
> > > > >> > > >> >
> > > > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > > >> > > >> > > Thanks Vincent for driving these, I have added my
> > comments
> > > to
> > > > >> the
> > > > >> > > AIP
> > > > >> > > >> too.
> > > > >> > > >> > >
> > > > >> > > >> > > Regards,
> > > > >> > > >> > > Kaxil
> > > > >> > > >> > >
> > > > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > > > (XC-AS/EAE-ADA-T)
> > > > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > > >> > > >> > >
> > > > >> > > >> > > > +1 on the comments of Vikram and Jarek, added main
> > points
> > > > on
> > > > >> > > >> confluence
> > > > >> > > >> > > >
> > > > >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > > >> > > >> > > > ________________________________
> > > > >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > > >> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > > >> > > >> > > > Subject: Re: [DISCUSS] External event driven
> scheduling
> > > in
> > > > >> Airflow
> > > > >> > > >> > > >
> > > > >> > > >> > > > Vincent,
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks for writing this up. The overview looks really
> > > good!
> > > > >> > > >> > > >
> > > > >> > > >> > > > I will leave my comments in the AIP as well, but at a
> > > high
> > > > >> level
> > > > >> > > >> they are
> > > > >> > > >> > > > both relatively focused on the "how", rather than the
> > > > "what".
> > > > >> > > >> > > > With respect to the pull / polling approach, I
> > completely
> > > > >> agree
> > > > >> > > >> that some
> > > > >> > > >> > > > incarnation of this is needed.
> > > > >> > > >> > > > I am less certain as to how on this part. The bespoke
> > > > >> triggerer
> > > > >> > > >> approach
> > > > >> > > >> > > > completely makes sense for the long tail here, but
> can
> > we
> > > > do
> > > > >> > > better
> > > > >> > > >> for the
> > > > >> > > >> > > > 20% of scenarios which cover well over 80% of usage
> > here
> > > is
> > > > >> the
> > > > >> > > >> question in
> > > > >> > > >> > > > my mind. Or, are you thinking of those as being
> covered
> > > in
> > > > >> the
> > > > >> > > >> "push"
> > > > >> > > >> > > > model?
> > > > >> > > >> > > >
> > > > >> > > >> > > > Which leads to the "push" model approach.
> > > > >> > > >> > > > I am struggling with the same question that Jarek
> > raised
> > > > here
> > > > >> > > about
> > > > >> > > >> whether
> > > > >> > > >> > > > we need a new Airflow entity over and beyond the
> > existing
> > > > >> REST API
> > > > >> > > >> for the
> > > > >> > > >> > > > same.
> > > > >> > > >> > > > I am concerned about this becoming a vector of attack
> > on
> > > > >> Airflow.
> > > > >> > > >> > > > I see that this is a hot topic of discussion in the
> > > > >> Confluence doc
> > > > >> > > >> as well,
> > > > >> > > >> > > > but wanted to summarize here as well, so it didn't
> get
> > > lost
> > > > >> in the
> > > > >> > > >> threads
> > > > >> > > >> > > > of comments.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Best regards,
> > > > >> > > >> > > > Vikram
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > > > >> jarek@potiuk.com>
> > > > >> > > >> wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > > Thanks Vincent. I took a look and I have a general
> > > > >> comment. I
> > > > >> > > >> > > > > strongly think external driven scheduling is really
> > > > needed
> > > > >> -
> > > > >> > > >> especially,
> > > > >> > > >> > > > it
> > > > >> > > >> > > > > should be much easier for a user to "plug-in" such
> an
> > > > >> external
> > > > >> > > >> event to
> > > > >> > > >> > > > > Airflow. And there are two parts of it - as
> correctly
> > > > >> stated
> > > > >> > > >> there - pull
> > > > >> > > >> > > > > and push.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > For the pull - I think it would be great to have a
> > kind
> > > > of
> > > > >> > > >> specialized
> > > > >> > > >> > > > > Triggers that will be started when DAG is parsed -
> > and
> > > > >> those
> > > > >> > > >> Triggers
> > > > >> > > >> > > > could
> > > > >> > > >> > > > > generate the events for DAGs. I think basically
> > that's
> > > > all
> > > > >> that
> > > > >> > > is
> > > > >> > > >> > > > needed,
> > > > >> > > >> > > > > for example I imagine a pubsub trigger that will
> > > > subscribe
> > > > >> to
> > > > >> > > >> messages
> > > > >> > > >> > > > > coming on the pubsub queue and fire "Asset" event
> > when
> > > a
> > > > >> message
> > > > >> > > >> is
> > > > >> > > >> > > > > received. Not much controversy there - I am not
> sure
> > > > about
> > > > >> the
> > > > >> > > >> polling
> > > > >> > > >> > > > > thing , because I've always believed that when
> > > > >> "asyncio-native"
> > > > >> > > >> Trigger
> > > > >> > > >> > > > is
> > > > >> > > >> > > > > run in the asyncio event loop, we do not "poll"
> every
> > > > >> second or
> > > > >> > > >> so (but
> > > > >> > > >> > > > > maybe this is just coming from some specific
> triggers
> > > > that
> > > > >> > > >> actually do
> > > > >> > > >> > > > > such regular poll. But yes - there are polls like
> > > > running
> > > > >> > > select
> > > > >> > > >> on the
> > > > >> > > >> > > > DB
> > > > >> > > >> > > > > that cannot be easily "async-ed" so having a
> > > configurable
> > > > >> > > polling
> > > > >> > > >> time
> > > > >> > > >> > > > > would be good there (but I am not sure maybe it's
> > even
> > > > >> possible
> > > > >> > > >> today). I
> > > > >> > > >> > > > > think this would be really great if we have that
> > > option,
> > > > >> because
> > > > >> > > >> it makes
> > > > >> > > >> > > > > it much easier to set up the authorization for
> > Airlfow
> > > > >> users -
> > > > >> > > >> rather
> > > > >> > > >> > > > than
> > > > >> > > >> > > > > setting up authorization and REST calls coming from
> > an
> > > > >> external
> > > > >> > > >> system,
> > > > >> > > >> > > > we
> > > > >> > > >> > > > > can utilize Connections of Airlfow to authorize
> such
> > a
> > > > >> Trigger
> > > > >> > > to
> > > > >> > > >> > > > subscribe
> > > > >> > > >> > > > > to events.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > For the push proposal - as I read the proposal,
> the
> > > main
> > > > >> point
> > > > >> > > >> behind it
> > > > >> > > >> > > > > is rather than users having to write "Airflow" way
> of
> > > > >> triggering
> > > > >> > > >> events
> > > > >> > > >> > > > and
> > > > >> > > >> > > > > configuring authentication (using REST API) to
> > generate
> > > > >> asset
> > > > >> > > >> events, is
> > > > >> > > >> > > > to
> > > > >> > > >> > > > > make Airflow natively understand external ways of
> > > pushing
> > > > >> - and
> > > > >> > > >> > > > effectively
> > > > >> > > >> > > > > authorizing and mapping such incoming unauthorized
> > > > >> requests into
> > > > >> > > >> event
> > > > >> > > >> > > > that
> > > > >> > > >> > > > > could be generated by an API REST call.
> > > > >> > > >> > > > > I am not really sure honestly if this is something
> > that
> > > > we
> > > > >> want
> > > > >> > > as
> > > > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such
> an
> > > > >> > > unauthorised
> > > > >> > > >> > > > endpoint
> > > > >> > > >> > > > > is probably not a good idea - for a variety of
> > reasons,
> > > > >> mostly
> > > > >> > > >> security.
> > > > >> > > >> > > > > And as I understand the goal is that users can
> easily
> > > > >> point at
> > > > >> > > >> > > > "3rd-party"
> > > > >> > > >> > > > > notification to Airflow and get the event
> generated.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > My feeling is that while this is needed - it should
> > be
> > > > >> > > >> externalised from
> > > > >> > > >> > > > > airlfow webserver. The authorization has to be set
> up
> > > > >> anyway
> > > > >> > > >> > > > additionally -
> > > > >> > > >> > > > > unlike in "poll" case - we cannot use Connections
> for
> > > > >> > > authorizing
> > > > >> > > >> > > > (because
> > > > >> > > >> > > > > it's not Airlfow that authorizes in an external
> > system
> > > -
> > > > >> it's
> > > > >> > > the
> > > > >> > > >> other
> > > > >> > > >> > > > way
> > > > >> > > >> > > > > round). So we have to anyhow setup "something
> extra"
> > in
> > > > >> Airflow
> > > > >> > > to
> > > > >> > > >> > > > > authorize the external system. Which could be what
> we
> > > > have
> > > > >> now -
> > > > >> > > >> user
> > > > >> > > >> > > > that
> > > > >> > > >> > > > > allows us to trigger the event. Which means that
> our
> > > REST
> > > > >> API
> > > > >> > > >> could
> > > > >> > > >> > > > > potentially be used the same way it is now, but we
> > will
> > > > >> need
> > > > >> > > >> "something"
> > > > >> > > >> > > > > (library, lambda function etc.) that users could
> > easily
> > > > >> setup in
> > > > >> > > >> the
> > > > >> > > >> > > > > external system to map whatever trigger they
> generate
> > > > >> natively
> > > > >> > > >> (say S3
> > > > >> > > >> > > > file
> > > > >> > > >> > > > > created) to Airflow REST API.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > As I see it - this is quite often used (and very
> > > > >> practical, that
> > > > >> > > >> you
> > > > >> > > >> > > > deploy
> > > > >> > > >> > > > > a cloud function or lambda that subscribes on the
> > event
> > > > >> received
> > > > >> > > >> when
> > > > >> > > >> > > > > S3/GCS is created. So it would be on the user to
> > deploy
> > > > >> such a
> > > > >> > > >> lambda -
> > > > >> > > >> > > > but
> > > > >> > > >> > > > > we **could** provide a library of those: say s3
> > lambda,
> > > > gcp
> > > > >> > > cloud
> > > > >> > > >> > > > function
> > > > >> > > >> > > > > in respective providers - with documentation how to
> > set
> > > > >> them up,
> > > > >> > > >> and how
> > > > >> > > >> > > > to
> > > > >> > > >> > > > > configure authorization and we would be generally
> > > "done".
> > > > >> I am
> > > > >> > > >> just not
> > > > >> > > >> > > > > sure if we need a new entity in Airflow for that
> > (Event
> > > > >> > > >> receiver). It
> > > > >> > > >> > > > feels
> > > > >> > > >> > > > > like it asks Airflow to take more responsibility,
> > when
> > > we
> > > > >> all
> > > > >> > > >> think on
> > > > >> > > >> > > > what
> > > > >> > > >> > > > > to "remove" from Airflow rather than "add" to it -
> > > > >> especially
> > > > >> > > >> when it
> > > > >> > > >> > > > comes
> > > > >> > > >> > > > > to external integrations. It feels to me that
> Airflow
> > > > >> should
> > > > >> > > make
> > > > >> > > >> it easy
> > > > >> > > >> > > > > to be triggered by such an external system and make
> > it
> > > > >> easy to
> > > > >> > > >> "map" to
> > > > >> > > >> > > > the
> > > > >> > > >> > > > > way we expect to get events triggered, but this
> > should
> > > be
> > > > >> done
> > > > >> > > >> outside of
> > > > >> > > >> > > > > Airflow. If the users can easily find in our docs
> > when
> > > > they
> > > > >> > > >> search "what
> > > > >> > > >> > > > do
> > > > >> > > >> > > > > I do to externally trigger Airflow on S3 change":
> > > either
> > > > a)
> > > > >> > > >> configure
> > > > >> > > >> > > > > polling in airflow using s3 Connection, or b)
> > "create a
> > > > >> user +
> > > > >> > > >> deploy
> > > > >> > > >> > > > this
> > > > >> > > >> > > > > lambda with those parameters" - that is "easy
> > enough"
> > > > and
> > > > >> very
> > > > >> > > >> practical
> > > > >> > > >> > > > > as well.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > But maybe I am not seeing the whole picture and the
> > > real
> > > > >> problem
> > > > >> > > >> it's
> > > > >> > > >> > > > > solving - so take it as a "first review pass" and
> > "guts
> > > > >> > > feeling".
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > J.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > > >> > > >> > > > <vincbeck@amazon.com.invalid
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > wrote:
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > > Hello everyone,
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > > I created a draft AIP regarding "External event
> > > driven
> > > > >> > > >> scheduling in
> > > > >> > > >> > > > > > Airflow". This proposal is about adding
> capability
> > in
> > > > >> Airflow
> > > > >> > > to
> > > > >> > > >> > > > schedule
> > > > >> > > >> > > > > > DAGs based on external events. Here are some
> > examples
> > > > of
> > > > >> such
> > > > >> > > >> external
> > > > >> > > >> > > > > > events:
> > > > >> > > >> > > > > > - A user signs up to one of the user pool defined
> > in
> > > my
> > > > >> cloud
> > > > >> > > >> provider
> > > > >> > > >> > > > > > - One of the databases used in my company has
> been
> > > > >> updated
> > > > >> > > >> > > > > > - A job in my cloud provider has been executed
> > > > >> successfully
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > > The intent of this AIP is to leverage datasets
> > (which
> > > > >> will be
> > > > >> > > >> soon
> > > > >> > > >> > > > > assets)
> > > > >> > > >> > > > > > and update them based on external events. I would
> > > like
> > > > to
> > > > >> > > >> propose this
> > > > >> > > >> > > > > AIP
> > > > >> > > >> > > > > > for discussion and more importantly, hear some
> > > > feedbacks
> > > > >> from
> > > > >> > > >> you :)
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > > >> > > >> > > > <
> > > > >> > > >> > > >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > > Vincent
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > >> > To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > >> > > >> > For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >> > > >> For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Ah, I didn't notice this discussion on the email thread, which I was
commenting on the wiki page.
I am comfortable with the poll-based approach, but struggling with the
push-based approach.
My concerns are generally consistent with the ones raised here.
I see the mapping/conversion function approach as being a way to avoid the
need for an additional component.
I am not really opposed to it, but concerned that it would come with a set
of challenges when in production for both scale and security.
On Thu, Aug 1, 2024 at 5:22 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I am all for it - if we want to stick to event bridge or similar and
> recommend it to our users, it's perfectly fine for me, It would be great
> however to add a documentation explaining the steps and some examples -
> ideally for most of our providers and "standard" ways of triggering such an
> event. This is even what I proposed originally whent the first version of
> the document was created ( to just document how to map the events
> externally).
>
> BTW. Yes - in this case you need to implement the logic in the
> event bridge, but the user will have to be added anyway (some kind of
> service account - because the API needs to be authorized - that part is not
> changed). (unless of course you want to use the same user for all kinds of
> external interfaces, which for security point of view is a very bad idea -
> each external system should have their own "service account" - that 's the
> best practices from the security point of view.
>
> J,
>
> On Fri, Aug 2, 2024 at 2:01 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > I was discussing this with Vincent. In either case, same as now or the
> one
> > proposed in the AIP, a user will have to use something like AWS
> > EventBridge[1] or GCS EventArc [2] where users will consume the event
> from
> > object storage (S3 object creation for example), and then they will have
> to
> > add the Airflow's Create Dataset endpoint to EventBridge [3]. Now, if you
> > just customize the payload to build the URI which is allowed (either via
> > GET / POST) in eventbridge, it works right now. However, with the current
> > proposal: a user will have to create a new user in Airflow and some
> mapping
> > to a function (that is either in the provider or a new user-defined
> > function) that can understand this specific payload, in this example the
> > payload for S3 events. This will become huge because this means that for
> > each payload, we will have to provide a new function and keep it updated.
> > From users POV, they will need to create a new user every time for a new
> > service (S3, Redshift, SNS, Bedrock etc). This will again likely have to
> go
> > to the Auth manager backend. Compared to what's available today -- i.e.
> > building a URI & extra metadata that can not only work with EventBridge
> or
> > Eventarc but by any service.
> >
> > Since we already have to use things like EventBridge or EventArc for
> > managed service providers to transform the event, it fits well with the
> > existing approach. AWS Blog [3] even has a similar example for Datadog
> > where they use input transformer "{"detail":"$.detail"}" before sending
> it
> > to Datadog's API.
> > >"Having producer of an event generating even in their standard way, make
> > it easy for airflow to consume it as a dataset event without external
> > entities":
> >
> >
> > [1]: https://aws.amazon.com/eventbridge/ |
> > https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
> > [2]: https://cloud.google.com/eventarc/docs
> > [3]:
> >
> >
> https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
> >
> > On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I proposed the mapping - because it's the easiest way (I think) to map
> > > between the "native" source to "airflow" target expectations. There are
> > > many producers of such events, and Airflow is the consumer. And it
> seems
> > > appropriate to have a way for our users to easily plug events produced
> > from
> > > one system into our "events" API - without having to employ external
> > > "mapper" (say lambda) doing the conversion. While I think it is indeed
> "a
> > > bit odd", it's a solution that might leverage most of what we have -
> > > authorisation and API exposure via "user" in API.
> > >
> > > While I - myself - find it it a bit unusual, I think it might do the
> job,
> > > But I wonder if there is any alternative solution to the problem of
> > "Having
> > > producer of an event generating even in their standard way, make it
> easy
> > > for airflow to consume it as a dataset event without external
> entities".
> > >
> > > On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > I would love for VOTE to get started on this one. I think most of the
> > > > commenters and those who replied to this email are happy with the
> > > proposal
> > > > on the poll-based approach.
> > > >
> > > > Regarding the push-based approach, I am not convinced that the
> proposed
> > > > implementation has any gains over what's already available with the
> > > Dataset
> > > > Event Create API; the one user-to-one function mapping is an odd user
> > > > experience. I'm curious to hear what others think.
> > > >
> > > > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com> wrote:
> > > >
> > > > > I agree with both of you that it is indeed a good idea and that it
> > can
> > > be
> > > > > added in Future work -- doesn't need to be part of this AIP.
> > > > >
> > > > > Thanks for the interest. I was not aware of such feature and this
> > looks
> > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > especially
> > > > >> for testing when you can easily replay events received in the
> past.
> > > > >> However, I do not think it should be part of the AIP and, as you
> > > > mentioned,
> > > > >> if should be a future work or a follow-up item of the AIP. Please
> > let
> > > me
> > > > >> know if you (or anyone) disagree with this and we can talk about
> it.
> > > > >> Otherwise I'll update the future work section of the AIP and
> mention
> > > > this
> > > > >> archive and replay feature.
> > > > >
> > > > >
> > > > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org>
> > wrote:
> > > > >
> > > > >> Hey Pavan,
> > > > >>
> > > > >> Thanks for the interest. I was not aware of such feature and this
> > > looks
> > > > >> really cool! I definitely think that can be useful for Airflow,
> > > > especially
> > > > >> for testing when you can easily replay events received in the
> past.
> > > > >> However, I do not think it should be part of the AIP and, as you
> > > > mentioned,
> > > > >> if should be a future work or a follow-up item of the AIP. Please
> > let
> > > me
> > > > >> know if you (or anyone) disagree with this and we can talk about
> it.
> > > > >> Otherwise I'll update the future work section of the AIP and
> mention
> > > > this
> > > > >> archive and replay feature.
> > > > >>
> > > > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > > > >> > Thanks Vincent, I took a look , this is really good. Don't have
> > > access
> > > > >> to
> > > > >> > the confluence page to comment :) so adding it here.
> > > > >> >
> > > > >> > As events arrive-->do somework-->end.
> > > > >> >
> > > > >> > So I'm uncertain if my comment pertains to the current poll/push
> > > model
> > > > >> or
> > > > >> > if it fits part of future work(seen event batching ).
> > > > >> >
> > > > >> > Have you given any thought to the event archival mechanism and
> > event
> > > > >> > replay? This could significantly aid in testing and recovery of
> > > > workflow
> > > > >> > and testing new functionality with events by just replay the
> > events.
> > > > The
> > > > >> > archival mechanism I am referring to is similar to today in AWS
> we
> > > > have
> > > > >> > Event Bridge Archive and Replay.
> > > > >> >
> > > > >> > Regards,
> > > > >> > Pavan
> > > > >> >
> > > > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > I actually did manage to take a look, thanks for the work. I
> am
> > +1
> > > > on
> > > > >> the
> > > > >> > > poll-based approach -- left a comment on the push-based: I am
> > not
> > > > >> sure of
> > > > >> > > why we need a function since create asset event API endpoint
> > > should
> > > > >> have
> > > > >> > > all info needed for what the Asset was.
> > > > >> > >
> > > > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > > > >> > > >
> > > > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <
> > vincbeck@apache.org
> > > >
> > > > >> wrote:
> > > > >> > > >
> > > > >> > > >> Hi everyone,
> > > > >> > > >>
> > > > >> > > >> I updated the AIP-82 given the different comments and
> > concerns
> > > I
> > > > >> > > >> received. I also tried to reply to all comments
> > individually. I
> > > > >> would
> > > > >> > > >> really appreciate if you can do a second pass and let me
> know
> > > > what
> > > > >> you
> > > > >> > > >> think. Overall, this is what I changed in the AIP:
> > > > >> > > >>
> > > > >> > > >> - Push based event-driven scheduling. I updated this
> section
> > > > >> entirely
> > > > >> > > >> because I received many concerns about the previous
> proposal.
> > > The
> > > > >> > > overall
> > > > >> > > >> idea now is to leverage the create asset event API endpoint
> > to
> > > > send
> > > > >> > > >> notifications from external (e.g. cloud provider) to
> Airflow
> > > > >> > > environment.
> > > > >> > > >>
> > > > >> > > >> - I updated the poll based event-driven scheduling DAG
> author
> > > > >> experience
> > > > >> > > >> to use a message queue scenario. I understood that this is
> > > > >> probably the
> > > > >> > > >> main use case we are trying to cover with this AIP, thus I
> > used
> > > > it
> > > > >> as
> > > > >> > > >> example and mentioned it multiple times across the AIP.
> > > > >> > > >>
> > > > >> > > >> Thanks again for your time :)
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > >> > > >>
> > > > >> > > >> Vincent
> > > > >> > > >>
> > > > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > > >> > > >> > Thanks a lot all for the comments, this is very much
> > > > >> appreciated! I
> > > > >> > > >> received many comments from this thread and in confluence,
> > > thanks
> > > > >> again.
> > > > >> > > >> I'll try to address them all in the AIP and will send an
> > email
> > > in
> > > > >> this
> > > > >> > > >> thread once done. I will most likely revisit the push-based
> > > > >> approach
> > > > >> > > given
> > > > >> > > >> the number of concerns I received, thanks Jarek for
> proposing
> > > > >> another
> > > > >> > > >> solution, I'll probably go down that path.
> > > > >> > > >> >
> > > > >> > > >> > One follow-up question Vikram.
> > > > >> > > >> >
> > > > >> > > >> > > The bespoke triggerer approach completely makes sense
> for
> > > the
> > > > >> long
> > > > >> > > >> tail here, but can we do better for the 20% of scenarios
> > which
> > > > >> cover
> > > > >> > > well
> > > > >> > > >> over 80% of usage here is the question in my mind. Or, are
> > you
> > > > >> thinking
> > > > >> > > of
> > > > >> > > >> those as being covered in the "push" model?
> > > > >> > > >> >
> > > > >> > > >> > Could you share more details about what is this "20% of
> > > > scenarios
> > > > >> > > which
> > > > >> > > >> cover well over 80% of usage" please?
> > > > >> > > >> >
> > > > >> > > >> > Vincent
> > > > >> > > >> >
> > > > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > > >> > > >> > > Thanks Vincent for driving these, I have added my
> > comments
> > > to
> > > > >> the
> > > > >> > > AIP
> > > > >> > > >> too.
> > > > >> > > >> > >
> > > > >> > > >> > > Regards,
> > > > >> > > >> > > Kaxil
> > > > >> > > >> > >
> > > > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > > > (XC-AS/EAE-ADA-T)
> > > > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > > >> > > >> > >
> > > > >> > > >> > > > +1 on the comments of Vikram and Jarek, added main
> > points
> > > > on
> > > > >> > > >> confluence
> > > > >> > > >> > > >
> > > > >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > > >> > > >> > > > ________________________________
> > > > >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > > >> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > > >> > > >> > > > Subject: Re: [DISCUSS] External event driven
> scheduling
> > > in
> > > > >> Airflow
> > > > >> > > >> > > >
> > > > >> > > >> > > > Vincent,
> > > > >> > > >> > > >
> > > > >> > > >> > > > Thanks for writing this up. The overview looks really
> > > good!
> > > > >> > > >> > > >
> > > > >> > > >> > > > I will leave my comments in the AIP as well, but at a
> > > high
> > > > >> level
> > > > >> > > >> they are
> > > > >> > > >> > > > both relatively focused on the "how", rather than the
> > > > "what".
> > > > >> > > >> > > > With respect to the pull / polling approach, I
> > completely
> > > > >> agree
> > > > >> > > >> that some
> > > > >> > > >> > > > incarnation of this is needed.
> > > > >> > > >> > > > I am less certain as to how on this part. The bespoke
> > > > >> triggerer
> > > > >> > > >> approach
> > > > >> > > >> > > > completely makes sense for the long tail here, but
> can
> > we
> > > > do
> > > > >> > > better
> > > > >> > > >> for the
> > > > >> > > >> > > > 20% of scenarios which cover well over 80% of usage
> > here
> > > is
> > > > >> the
> > > > >> > > >> question in
> > > > >> > > >> > > > my mind. Or, are you thinking of those as being
> covered
> > > in
> > > > >> the
> > > > >> > > >> "push"
> > > > >> > > >> > > > model?
> > > > >> > > >> > > >
> > > > >> > > >> > > > Which leads to the "push" model approach.
> > > > >> > > >> > > > I am struggling with the same question that Jarek
> > raised
> > > > here
> > > > >> > > about
> > > > >> > > >> whether
> > > > >> > > >> > > > we need a new Airflow entity over and beyond the
> > existing
> > > > >> REST API
> > > > >> > > >> for the
> > > > >> > > >> > > > same.
> > > > >> > > >> > > > I am concerned about this becoming a vector of attack
> > on
> > > > >> Airflow.
> > > > >> > > >> > > > I see that this is a hot topic of discussion in the
> > > > >> Confluence doc
> > > > >> > > >> as well,
> > > > >> > > >> > > > but wanted to summarize here as well, so it didn't
> get
> > > lost
> > > > >> in the
> > > > >> > > >> threads
> > > > >> > > >> > > > of comments.
> > > > >> > > >> > > >
> > > > >> > > >> > > > Best regards,
> > > > >> > > >> > > > Vikram
> > > > >> > > >> > > >
> > > > >> > > >> > > >
> > > > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > > > >> jarek@potiuk.com>
> > > > >> > > >> wrote:
> > > > >> > > >> > > >
> > > > >> > > >> > > > > Thanks Vincent. I took a look and I have a general
> > > > >> comment. I
> > > > >> > > >> > > > > strongly think external driven scheduling is really
> > > > needed
> > > > >> -
> > > > >> > > >> especially,
> > > > >> > > >> > > > it
> > > > >> > > >> > > > > should be much easier for a user to "plug-in" such
> an
> > > > >> external
> > > > >> > > >> event to
> > > > >> > > >> > > > > Airflow. And there are two parts of it - as
> correctly
> > > > >> stated
> > > > >> > > >> there - pull
> > > > >> > > >> > > > > and push.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > For the pull - I think it would be great to have a
> > kind
> > > > of
> > > > >> > > >> specialized
> > > > >> > > >> > > > > Triggers that will be started when DAG is parsed -
> > and
> > > > >> those
> > > > >> > > >> Triggers
> > > > >> > > >> > > > could
> > > > >> > > >> > > > > generate the events for DAGs. I think basically
> > that's
> > > > all
> > > > >> that
> > > > >> > > is
> > > > >> > > >> > > > needed,
> > > > >> > > >> > > > > for example I imagine a pubsub trigger that will
> > > > subscribe
> > > > >> to
> > > > >> > > >> messages
> > > > >> > > >> > > > > coming on the pubsub queue and fire "Asset" event
> > when
> > > a
> > > > >> message
> > > > >> > > >> is
> > > > >> > > >> > > > > received. Not much controversy there - I am not
> sure
> > > > about
> > > > >> the
> > > > >> > > >> polling
> > > > >> > > >> > > > > thing , because I've always believed that when
> > > > >> "asyncio-native"
> > > > >> > > >> Trigger
> > > > >> > > >> > > > is
> > > > >> > > >> > > > > run in the asyncio event loop, we do not "poll"
> every
> > > > >> second or
> > > > >> > > >> so (but
> > > > >> > > >> > > > > maybe this is just coming from some specific
> triggers
> > > > that
> > > > >> > > >> actually do
> > > > >> > > >> > > > > such regular poll. But yes - there are polls like
> > > > running
> > > > >> > > select
> > > > >> > > >> on the
> > > > >> > > >> > > > DB
> > > > >> > > >> > > > > that cannot be easily "async-ed" so having a
> > > configurable
> > > > >> > > polling
> > > > >> > > >> time
> > > > >> > > >> > > > > would be good there (but I am not sure maybe it's
> > even
> > > > >> possible
> > > > >> > > >> today). I
> > > > >> > > >> > > > > think this would be really great if we have that
> > > option,
> > > > >> because
> > > > >> > > >> it makes
> > > > >> > > >> > > > > it much easier to set up the authorization for
> > Airlfow
> > > > >> users -
> > > > >> > > >> rather
> > > > >> > > >> > > > than
> > > > >> > > >> > > > > setting up authorization and REST calls coming from
> > an
> > > > >> external
> > > > >> > > >> system,
> > > > >> > > >> > > > we
> > > > >> > > >> > > > > can utilize Connections of Airlfow to authorize
> such
> > a
> > > > >> Trigger
> > > > >> > > to
> > > > >> > > >> > > > subscribe
> > > > >> > > >> > > > > to events.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > For the push proposal - as I read the proposal,
> the
> > > main
> > > > >> point
> > > > >> > > >> behind it
> > > > >> > > >> > > > > is rather than users having to write "Airflow" way
> of
> > > > >> triggering
> > > > >> > > >> events
> > > > >> > > >> > > > and
> > > > >> > > >> > > > > configuring authentication (using REST API) to
> > generate
> > > > >> asset
> > > > >> > > >> events, is
> > > > >> > > >> > > > to
> > > > >> > > >> > > > > make Airflow natively understand external ways of
> > > pushing
> > > > >> - and
> > > > >> > > >> > > > effectively
> > > > >> > > >> > > > > authorizing and mapping such incoming unauthorized
> > > > >> requests into
> > > > >> > > >> event
> > > > >> > > >> > > > that
> > > > >> > > >> > > > > could be generated by an API REST call.
> > > > >> > > >> > > > > I am not really sure honestly if this is something
> > that
> > > > we
> > > > >> want
> > > > >> > > as
> > > > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such
> an
> > > > >> > > unauthorised
> > > > >> > > >> > > > endpoint
> > > > >> > > >> > > > > is probably not a good idea - for a variety of
> > reasons,
> > > > >> mostly
> > > > >> > > >> security.
> > > > >> > > >> > > > > And as I understand the goal is that users can
> easily
> > > > >> point at
> > > > >> > > >> > > > "3rd-party"
> > > > >> > > >> > > > > notification to Airflow and get the event
> generated.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > My feeling is that while this is needed - it should
> > be
> > > > >> > > >> externalised from
> > > > >> > > >> > > > > airlfow webserver. The authorization has to be set
> up
> > > > >> anyway
> > > > >> > > >> > > > additionally -
> > > > >> > > >> > > > > unlike in "poll" case - we cannot use Connections
> for
> > > > >> > > authorizing
> > > > >> > > >> > > > (because
> > > > >> > > >> > > > > it's not Airlfow that authorizes in an external
> > system
> > > -
> > > > >> it's
> > > > >> > > the
> > > > >> > > >> other
> > > > >> > > >> > > > way
> > > > >> > > >> > > > > round). So we have to anyhow setup "something
> extra"
> > in
> > > > >> Airflow
> > > > >> > > to
> > > > >> > > >> > > > > authorize the external system. Which could be what
> we
> > > > have
> > > > >> now -
> > > > >> > > >> user
> > > > >> > > >> > > > that
> > > > >> > > >> > > > > allows us to trigger the event. Which means that
> our
> > > REST
> > > > >> API
> > > > >> > > >> could
> > > > >> > > >> > > > > potentially be used the same way it is now, but we
> > will
> > > > >> need
> > > > >> > > >> "something"
> > > > >> > > >> > > > > (library, lambda function etc.) that users could
> > easily
> > > > >> setup in
> > > > >> > > >> the
> > > > >> > > >> > > > > external system to map whatever trigger they
> generate
> > > > >> natively
> > > > >> > > >> (say S3
> > > > >> > > >> > > > file
> > > > >> > > >> > > > > created) to Airflow REST API.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > As I see it - this is quite often used (and very
> > > > >> practical, that
> > > > >> > > >> you
> > > > >> > > >> > > > deploy
> > > > >> > > >> > > > > a cloud function or lambda that subscribes on the
> > event
> > > > >> received
> > > > >> > > >> when
> > > > >> > > >> > > > > S3/GCS is created. So it would be on the user to
> > deploy
> > > > >> such a
> > > > >> > > >> lambda -
> > > > >> > > >> > > > but
> > > > >> > > >> > > > > we **could** provide a library of those: say s3
> > lambda,
> > > > gcp
> > > > >> > > cloud
> > > > >> > > >> > > > function
> > > > >> > > >> > > > > in respective providers - with documentation how to
> > set
> > > > >> them up,
> > > > >> > > >> and how
> > > > >> > > >> > > > to
> > > > >> > > >> > > > > configure authorization and we would be generally
> > > "done".
> > > > >> I am
> > > > >> > > >> just not
> > > > >> > > >> > > > > sure if we need a new entity in Airflow for that
> > (Event
> > > > >> > > >> receiver). It
> > > > >> > > >> > > > feels
> > > > >> > > >> > > > > like it asks Airflow to take more responsibility,
> > when
> > > we
> > > > >> all
> > > > >> > > >> think on
> > > > >> > > >> > > > what
> > > > >> > > >> > > > > to "remove" from Airflow rather than "add" to it -
> > > > >> especially
> > > > >> > > >> when it
> > > > >> > > >> > > > comes
> > > > >> > > >> > > > > to external integrations. It feels to me that
> Airflow
> > > > >> should
> > > > >> > > make
> > > > >> > > >> it easy
> > > > >> > > >> > > > > to be triggered by such an external system and make
> > it
> > > > >> easy to
> > > > >> > > >> "map" to
> > > > >> > > >> > > > the
> > > > >> > > >> > > > > way we expect to get events triggered, but this
> > should
> > > be
> > > > >> done
> > > > >> > > >> outside of
> > > > >> > > >> > > > > Airflow. If the users can easily find in our docs
> > when
> > > > they
> > > > >> > > >> search "what
> > > > >> > > >> > > > do
> > > > >> > > >> > > > > I do to externally trigger Airflow on S3 change":
> > > either
> > > > a)
> > > > >> > > >> configure
> > > > >> > > >> > > > > polling in airflow using s3 Connection, or b)
> > "create a
> > > > >> user +
> > > > >> > > >> deploy
> > > > >> > > >> > > > this
> > > > >> > > >> > > > > lambda with those parameters" - that is "easy
> > enough"
> > > > and
> > > > >> very
> > > > >> > > >> practical
> > > > >> > > >> > > > > as well.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > But maybe I am not seeing the whole picture and the
> > > real
> > > > >> problem
> > > > >> > > >> it's
> > > > >> > > >> > > > > solving - so take it as a "first review pass" and
> > "guts
> > > > >> > > feeling".
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > J.
> > > > >> > > >> > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > > >> > > >> > > > <vincbeck@amazon.com.invalid
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > wrote:
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > > Hello everyone,
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > > I created a draft AIP regarding "External event
> > > driven
> > > > >> > > >> scheduling in
> > > > >> > > >> > > > > > Airflow". This proposal is about adding
> capability
> > in
> > > > >> Airflow
> > > > >> > > to
> > > > >> > > >> > > > schedule
> > > > >> > > >> > > > > > DAGs based on external events. Here are some
> > examples
> > > > of
> > > > >> such
> > > > >> > > >> external
> > > > >> > > >> > > > > > events:
> > > > >> > > >> > > > > > - A user signs up to one of the user pool defined
> > in
> > > my
> > > > >> cloud
> > > > >> > > >> provider
> > > > >> > > >> > > > > > - One of the databases used in my company has
> been
> > > > >> updated
> > > > >> > > >> > > > > > - A job in my cloud provider has been executed
> > > > >> successfully
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > > The intent of this AIP is to leverage datasets
> > (which
> > > > >> will be
> > > > >> > > >> soon
> > > > >> > > >> > > > > assets)
> > > > >> > > >> > > > > > and update them based on external events. I would
> > > like
> > > > to
> > > > >> > > >> propose this
> > > > >> > > >> > > > > AIP
> > > > >> > > >> > > > > > for discussion and more importantly, hear some
> > > > feedbacks
> > > > >> from
> > > > >> > > >> you :)
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > > >> > > >> > > > <
> > > > >> > > >> > > >
> > > > >> > > >>
> > > > >> > >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > >> > > >> > > > >
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > > > Vincent
> > > > >> > > >> > > > > >
> > > > >> > > >> > > > >
> > > > >> > > >> > > >
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > >> > To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > >> > > >> > For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >> > > >> For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > >> > > >>
> > > > >> > > >>
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
I am all for it - if we want to stick to event bridge or similar and
recommend it to our users, it's perfectly fine for me, It would be great
however to add a documentation explaining the steps and some examples -
ideally for most of our providers and "standard" ways of triggering such an
event. This is even what I proposed originally whent the first version of
the document was created ( to just document how to map the events
externally).
BTW. Yes - in this case you need to implement the logic in the
event bridge, but the user will have to be added anyway (some kind of
service account - because the API needs to be authorized - that part is not
changed). (unless of course you want to use the same user for all kinds of
external interfaces, which for security point of view is a very bad idea -
each external system should have their own "service account" - that 's the
best practices from the security point of view.
J,
On Fri, Aug 2, 2024 at 2:01 AM Kaxil Naik <ka...@gmail.com> wrote:
> I was discussing this with Vincent. In either case, same as now or the one
> proposed in the AIP, a user will have to use something like AWS
> EventBridge[1] or GCS EventArc [2] where users will consume the event from
> object storage (S3 object creation for example), and then they will have to
> add the Airflow's Create Dataset endpoint to EventBridge [3]. Now, if you
> just customize the payload to build the URI which is allowed (either via
> GET / POST) in eventbridge, it works right now. However, with the current
> proposal: a user will have to create a new user in Airflow and some mapping
> to a function (that is either in the provider or a new user-defined
> function) that can understand this specific payload, in this example the
> payload for S3 events. This will become huge because this means that for
> each payload, we will have to provide a new function and keep it updated.
> From users POV, they will need to create a new user every time for a new
> service (S3, Redshift, SNS, Bedrock etc). This will again likely have to go
> to the Auth manager backend. Compared to what's available today -- i.e.
> building a URI & extra metadata that can not only work with EventBridge or
> Eventarc but by any service.
>
> Since we already have to use things like EventBridge or EventArc for
> managed service providers to transform the event, it fits well with the
> existing approach. AWS Blog [3] even has a similar example for Datadog
> where they use input transformer "{"detail":"$.detail"}" before sending it
> to Datadog's API.
> >"Having producer of an event generating even in their standard way, make
> it easy for airflow to consume it as a dataset event without external
> entities":
>
>
> [1]: https://aws.amazon.com/eventbridge/ |
> https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
> [2]: https://cloud.google.com/eventarc/docs
> [3]:
>
> https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
>
> On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I proposed the mapping - because it's the easiest way (I think) to map
> > between the "native" source to "airflow" target expectations. There are
> > many producers of such events, and Airflow is the consumer. And it seems
> > appropriate to have a way for our users to easily plug events produced
> from
> > one system into our "events" API - without having to employ external
> > "mapper" (say lambda) doing the conversion. While I think it is indeed "a
> > bit odd", it's a solution that might leverage most of what we have -
> > authorisation and API exposure via "user" in API.
> >
> > While I - myself - find it it a bit unusual, I think it might do the job,
> > But I wonder if there is any alternative solution to the problem of
> "Having
> > producer of an event generating even in their standard way, make it easy
> > for airflow to consume it as a dataset event without external entities".
> >
> > On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > I would love for VOTE to get started on this one. I think most of the
> > > commenters and those who replied to this email are happy with the
> > proposal
> > > on the poll-based approach.
> > >
> > > Regarding the push-based approach, I am not convinced that the proposed
> > > implementation has any gains over what's already available with the
> > Dataset
> > > Event Create API; the one user-to-one function mapping is an odd user
> > > experience. I'm curious to hear what others think.
> > >
> > > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > I agree with both of you that it is indeed a good idea and that it
> can
> > be
> > > > added in Future work -- doesn't need to be part of this AIP.
> > > >
> > > > Thanks for the interest. I was not aware of such feature and this
> looks
> > > >> really cool! I definitely think that can be useful for Airflow,
> > > especially
> > > >> for testing when you can easily replay events received in the past.
> > > >> However, I do not think it should be part of the AIP and, as you
> > > mentioned,
> > > >> if should be a future work or a follow-up item of the AIP. Please
> let
> > me
> > > >> know if you (or anyone) disagree with this and we can talk about it.
> > > >> Otherwise I'll update the future work section of the AIP and mention
> > > this
> > > >> archive and replay feature.
> > > >
> > > >
> > > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org>
> wrote:
> > > >
> > > >> Hey Pavan,
> > > >>
> > > >> Thanks for the interest. I was not aware of such feature and this
> > looks
> > > >> really cool! I definitely think that can be useful for Airflow,
> > > especially
> > > >> for testing when you can easily replay events received in the past.
> > > >> However, I do not think it should be part of the AIP and, as you
> > > mentioned,
> > > >> if should be a future work or a follow-up item of the AIP. Please
> let
> > me
> > > >> know if you (or anyone) disagree with this and we can talk about it.
> > > >> Otherwise I'll update the future work section of the AIP and mention
> > > this
> > > >> archive and replay feature.
> > > >>
> > > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > > >> > Thanks Vincent, I took a look , this is really good. Don't have
> > access
> > > >> to
> > > >> > the confluence page to comment :) so adding it here.
> > > >> >
> > > >> > As events arrive-->do somework-->end.
> > > >> >
> > > >> > So I'm uncertain if my comment pertains to the current poll/push
> > model
> > > >> or
> > > >> > if it fits part of future work(seen event batching ).
> > > >> >
> > > >> > Have you given any thought to the event archival mechanism and
> event
> > > >> > replay? This could significantly aid in testing and recovery of
> > > workflow
> > > >> > and testing new functionality with events by just replay the
> events.
> > > The
> > > >> > archival mechanism I am referring to is similar to today in AWS we
> > > have
> > > >> > Event Bridge Archive and Replay.
> > > >> >
> > > >> > Regards,
> > > >> > Pavan
> > > >> >
> > > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > I actually did manage to take a look, thanks for the work. I am
> +1
> > > on
> > > >> the
> > > >> > > poll-based approach -- left a comment on the push-based: I am
> not
> > > >> sure of
> > > >> > > why we need a function since create asset event API endpoint
> > should
> > > >> have
> > > >> > > all info needed for what the Asset was.
> > > >> > >
> > > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com>
> > > wrote:
> > > >> > >
> > > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > > >> > > >
> > > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <
> vincbeck@apache.org
> > >
> > > >> wrote:
> > > >> > > >
> > > >> > > >> Hi everyone,
> > > >> > > >>
> > > >> > > >> I updated the AIP-82 given the different comments and
> concerns
> > I
> > > >> > > >> received. I also tried to reply to all comments
> individually. I
> > > >> would
> > > >> > > >> really appreciate if you can do a second pass and let me know
> > > what
> > > >> you
> > > >> > > >> think. Overall, this is what I changed in the AIP:
> > > >> > > >>
> > > >> > > >> - Push based event-driven scheduling. I updated this section
> > > >> entirely
> > > >> > > >> because I received many concerns about the previous proposal.
> > The
> > > >> > > overall
> > > >> > > >> idea now is to leverage the create asset event API endpoint
> to
> > > send
> > > >> > > >> notifications from external (e.g. cloud provider) to Airflow
> > > >> > > environment.
> > > >> > > >>
> > > >> > > >> - I updated the poll based event-driven scheduling DAG author
> > > >> experience
> > > >> > > >> to use a message queue scenario. I understood that this is
> > > >> probably the
> > > >> > > >> main use case we are trying to cover with this AIP, thus I
> used
> > > it
> > > >> as
> > > >> > > >> example and mentioned it multiple times across the AIP.
> > > >> > > >>
> > > >> > > >> Thanks again for your time :)
> > > >> > > >>
> > > >> > > >>
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > >> > > >>
> > > >> > > >> Vincent
> > > >> > > >>
> > > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > >> > > >> > Thanks a lot all for the comments, this is very much
> > > >> appreciated! I
> > > >> > > >> received many comments from this thread and in confluence,
> > thanks
> > > >> again.
> > > >> > > >> I'll try to address them all in the AIP and will send an
> email
> > in
> > > >> this
> > > >> > > >> thread once done. I will most likely revisit the push-based
> > > >> approach
> > > >> > > given
> > > >> > > >> the number of concerns I received, thanks Jarek for proposing
> > > >> another
> > > >> > > >> solution, I'll probably go down that path.
> > > >> > > >> >
> > > >> > > >> > One follow-up question Vikram.
> > > >> > > >> >
> > > >> > > >> > > The bespoke triggerer approach completely makes sense for
> > the
> > > >> long
> > > >> > > >> tail here, but can we do better for the 20% of scenarios
> which
> > > >> cover
> > > >> > > well
> > > >> > > >> over 80% of usage here is the question in my mind. Or, are
> you
> > > >> thinking
> > > >> > > of
> > > >> > > >> those as being covered in the "push" model?
> > > >> > > >> >
> > > >> > > >> > Could you share more details about what is this "20% of
> > > scenarios
> > > >> > > which
> > > >> > > >> cover well over 80% of usage" please?
> > > >> > > >> >
> > > >> > > >> > Vincent
> > > >> > > >> >
> > > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > >> > > >> > > Thanks Vincent for driving these, I have added my
> comments
> > to
> > > >> the
> > > >> > > AIP
> > > >> > > >> too.
> > > >> > > >> > >
> > > >> > > >> > > Regards,
> > > >> > > >> > > Kaxil
> > > >> > > >> > >
> > > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > > (XC-AS/EAE-ADA-T)
> > > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > >> > > >> > >
> > > >> > > >> > > > +1 on the comments of Vikram and Jarek, added main
> points
> > > on
> > > >> > > >> confluence
> > > >> > > >> > > >
> > > >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > >> > > >> > > > ________________________________
> > > >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > >> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > >> > > >> > > > Subject: Re: [DISCUSS] External event driven scheduling
> > in
> > > >> Airflow
> > > >> > > >> > > >
> > > >> > > >> > > > Vincent,
> > > >> > > >> > > >
> > > >> > > >> > > > Thanks for writing this up. The overview looks really
> > good!
> > > >> > > >> > > >
> > > >> > > >> > > > I will leave my comments in the AIP as well, but at a
> > high
> > > >> level
> > > >> > > >> they are
> > > >> > > >> > > > both relatively focused on the "how", rather than the
> > > "what".
> > > >> > > >> > > > With respect to the pull / polling approach, I
> completely
> > > >> agree
> > > >> > > >> that some
> > > >> > > >> > > > incarnation of this is needed.
> > > >> > > >> > > > I am less certain as to how on this part. The bespoke
> > > >> triggerer
> > > >> > > >> approach
> > > >> > > >> > > > completely makes sense for the long tail here, but can
> we
> > > do
> > > >> > > better
> > > >> > > >> for the
> > > >> > > >> > > > 20% of scenarios which cover well over 80% of usage
> here
> > is
> > > >> the
> > > >> > > >> question in
> > > >> > > >> > > > my mind. Or, are you thinking of those as being covered
> > in
> > > >> the
> > > >> > > >> "push"
> > > >> > > >> > > > model?
> > > >> > > >> > > >
> > > >> > > >> > > > Which leads to the "push" model approach.
> > > >> > > >> > > > I am struggling with the same question that Jarek
> raised
> > > here
> > > >> > > about
> > > >> > > >> whether
> > > >> > > >> > > > we need a new Airflow entity over and beyond the
> existing
> > > >> REST API
> > > >> > > >> for the
> > > >> > > >> > > > same.
> > > >> > > >> > > > I am concerned about this becoming a vector of attack
> on
> > > >> Airflow.
> > > >> > > >> > > > I see that this is a hot topic of discussion in the
> > > >> Confluence doc
> > > >> > > >> as well,
> > > >> > > >> > > > but wanted to summarize here as well, so it didn't get
> > lost
> > > >> in the
> > > >> > > >> threads
> > > >> > > >> > > > of comments.
> > > >> > > >> > > >
> > > >> > > >> > > > Best regards,
> > > >> > > >> > > > Vikram
> > > >> > > >> > > >
> > > >> > > >> > > >
> > > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > > >> jarek@potiuk.com>
> > > >> > > >> wrote:
> > > >> > > >> > > >
> > > >> > > >> > > > > Thanks Vincent. I took a look and I have a general
> > > >> comment. I
> > > >> > > >> > > > > strongly think external driven scheduling is really
> > > needed
> > > >> -
> > > >> > > >> especially,
> > > >> > > >> > > > it
> > > >> > > >> > > > > should be much easier for a user to "plug-in" such an
> > > >> external
> > > >> > > >> event to
> > > >> > > >> > > > > Airflow. And there are two parts of it - as correctly
> > > >> stated
> > > >> > > >> there - pull
> > > >> > > >> > > > > and push.
> > > >> > > >> > > > >
> > > >> > > >> > > > > For the pull - I think it would be great to have a
> kind
> > > of
> > > >> > > >> specialized
> > > >> > > >> > > > > Triggers that will be started when DAG is parsed -
> and
> > > >> those
> > > >> > > >> Triggers
> > > >> > > >> > > > could
> > > >> > > >> > > > > generate the events for DAGs. I think basically
> that's
> > > all
> > > >> that
> > > >> > > is
> > > >> > > >> > > > needed,
> > > >> > > >> > > > > for example I imagine a pubsub trigger that will
> > > subscribe
> > > >> to
> > > >> > > >> messages
> > > >> > > >> > > > > coming on the pubsub queue and fire "Asset" event
> when
> > a
> > > >> message
> > > >> > > >> is
> > > >> > > >> > > > > received. Not much controversy there - I am not sure
> > > about
> > > >> the
> > > >> > > >> polling
> > > >> > > >> > > > > thing , because I've always believed that when
> > > >> "asyncio-native"
> > > >> > > >> Trigger
> > > >> > > >> > > > is
> > > >> > > >> > > > > run in the asyncio event loop, we do not "poll" every
> > > >> second or
> > > >> > > >> so (but
> > > >> > > >> > > > > maybe this is just coming from some specific triggers
> > > that
> > > >> > > >> actually do
> > > >> > > >> > > > > such regular poll. But yes - there are polls like
> > > running
> > > >> > > select
> > > >> > > >> on the
> > > >> > > >> > > > DB
> > > >> > > >> > > > > that cannot be easily "async-ed" so having a
> > configurable
> > > >> > > polling
> > > >> > > >> time
> > > >> > > >> > > > > would be good there (but I am not sure maybe it's
> even
> > > >> possible
> > > >> > > >> today). I
> > > >> > > >> > > > > think this would be really great if we have that
> > option,
> > > >> because
> > > >> > > >> it makes
> > > >> > > >> > > > > it much easier to set up the authorization for
> Airlfow
> > > >> users -
> > > >> > > >> rather
> > > >> > > >> > > > than
> > > >> > > >> > > > > setting up authorization and REST calls coming from
> an
> > > >> external
> > > >> > > >> system,
> > > >> > > >> > > > we
> > > >> > > >> > > > > can utilize Connections of Airlfow to authorize such
> a
> > > >> Trigger
> > > >> > > to
> > > >> > > >> > > > subscribe
> > > >> > > >> > > > > to events.
> > > >> > > >> > > > >
> > > >> > > >> > > > > For the push proposal - as I read the proposal, the
> > main
> > > >> point
> > > >> > > >> behind it
> > > >> > > >> > > > > is rather than users having to write "Airflow" way of
> > > >> triggering
> > > >> > > >> events
> > > >> > > >> > > > and
> > > >> > > >> > > > > configuring authentication (using REST API) to
> generate
> > > >> asset
> > > >> > > >> events, is
> > > >> > > >> > > > to
> > > >> > > >> > > > > make Airflow natively understand external ways of
> > pushing
> > > >> - and
> > > >> > > >> > > > effectively
> > > >> > > >> > > > > authorizing and mapping such incoming unauthorized
> > > >> requests into
> > > >> > > >> event
> > > >> > > >> > > > that
> > > >> > > >> > > > > could be generated by an API REST call.
> > > >> > > >> > > > > I am not really sure honestly if this is something
> that
> > > we
> > > >> want
> > > >> > > as
> > > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such an
> > > >> > > unauthorised
> > > >> > > >> > > > endpoint
> > > >> > > >> > > > > is probably not a good idea - for a variety of
> reasons,
> > > >> mostly
> > > >> > > >> security.
> > > >> > > >> > > > > And as I understand the goal is that users can easily
> > > >> point at
> > > >> > > >> > > > "3rd-party"
> > > >> > > >> > > > > notification to Airflow and get the event generated.
> > > >> > > >> > > > >
> > > >> > > >> > > > > My feeling is that while this is needed - it should
> be
> > > >> > > >> externalised from
> > > >> > > >> > > > > airlfow webserver. The authorization has to be set up
> > > >> anyway
> > > >> > > >> > > > additionally -
> > > >> > > >> > > > > unlike in "poll" case - we cannot use Connections for
> > > >> > > authorizing
> > > >> > > >> > > > (because
> > > >> > > >> > > > > it's not Airlfow that authorizes in an external
> system
> > -
> > > >> it's
> > > >> > > the
> > > >> > > >> other
> > > >> > > >> > > > way
> > > >> > > >> > > > > round). So we have to anyhow setup "something extra"
> in
> > > >> Airflow
> > > >> > > to
> > > >> > > >> > > > > authorize the external system. Which could be what we
> > > have
> > > >> now -
> > > >> > > >> user
> > > >> > > >> > > > that
> > > >> > > >> > > > > allows us to trigger the event. Which means that our
> > REST
> > > >> API
> > > >> > > >> could
> > > >> > > >> > > > > potentially be used the same way it is now, but we
> will
> > > >> need
> > > >> > > >> "something"
> > > >> > > >> > > > > (library, lambda function etc.) that users could
> easily
> > > >> setup in
> > > >> > > >> the
> > > >> > > >> > > > > external system to map whatever trigger they generate
> > > >> natively
> > > >> > > >> (say S3
> > > >> > > >> > > > file
> > > >> > > >> > > > > created) to Airflow REST API.
> > > >> > > >> > > > >
> > > >> > > >> > > > > As I see it - this is quite often used (and very
> > > >> practical, that
> > > >> > > >> you
> > > >> > > >> > > > deploy
> > > >> > > >> > > > > a cloud function or lambda that subscribes on the
> event
> > > >> received
> > > >> > > >> when
> > > >> > > >> > > > > S3/GCS is created. So it would be on the user to
> deploy
> > > >> such a
> > > >> > > >> lambda -
> > > >> > > >> > > > but
> > > >> > > >> > > > > we **could** provide a library of those: say s3
> lambda,
> > > gcp
> > > >> > > cloud
> > > >> > > >> > > > function
> > > >> > > >> > > > > in respective providers - with documentation how to
> set
> > > >> them up,
> > > >> > > >> and how
> > > >> > > >> > > > to
> > > >> > > >> > > > > configure authorization and we would be generally
> > "done".
> > > >> I am
> > > >> > > >> just not
> > > >> > > >> > > > > sure if we need a new entity in Airflow for that
> (Event
> > > >> > > >> receiver). It
> > > >> > > >> > > > feels
> > > >> > > >> > > > > like it asks Airflow to take more responsibility,
> when
> > we
> > > >> all
> > > >> > > >> think on
> > > >> > > >> > > > what
> > > >> > > >> > > > > to "remove" from Airflow rather than "add" to it -
> > > >> especially
> > > >> > > >> when it
> > > >> > > >> > > > comes
> > > >> > > >> > > > > to external integrations. It feels to me that Airflow
> > > >> should
> > > >> > > make
> > > >> > > >> it easy
> > > >> > > >> > > > > to be triggered by such an external system and make
> it
> > > >> easy to
> > > >> > > >> "map" to
> > > >> > > >> > > > the
> > > >> > > >> > > > > way we expect to get events triggered, but this
> should
> > be
> > > >> done
> > > >> > > >> outside of
> > > >> > > >> > > > > Airflow. If the users can easily find in our docs
> when
> > > they
> > > >> > > >> search "what
> > > >> > > >> > > > do
> > > >> > > >> > > > > I do to externally trigger Airflow on S3 change":
> > either
> > > a)
> > > >> > > >> configure
> > > >> > > >> > > > > polling in airflow using s3 Connection, or b)
> "create a
> > > >> user +
> > > >> > > >> deploy
> > > >> > > >> > > > this
> > > >> > > >> > > > > lambda with those parameters" - that is "easy
> enough"
> > > and
> > > >> very
> > > >> > > >> practical
> > > >> > > >> > > > > as well.
> > > >> > > >> > > > >
> > > >> > > >> > > > > But maybe I am not seeing the whole picture and the
> > real
> > > >> problem
> > > >> > > >> it's
> > > >> > > >> > > > > solving - so take it as a "first review pass" and
> "guts
> > > >> > > feeling".
> > > >> > > >> > > > >
> > > >> > > >> > > > > J.
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > >> > > >> > > > <vincbeck@amazon.com.invalid
> > > >> > > >> > > > > >
> > > >> > > >> > > > > wrote:
> > > >> > > >> > > > >
> > > >> > > >> > > > > > Hello everyone,
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > I created a draft AIP regarding "External event
> > driven
> > > >> > > >> scheduling in
> > > >> > > >> > > > > > Airflow". This proposal is about adding capability
> in
> > > >> Airflow
> > > >> > > to
> > > >> > > >> > > > schedule
> > > >> > > >> > > > > > DAGs based on external events. Here are some
> examples
> > > of
> > > >> such
> > > >> > > >> external
> > > >> > > >> > > > > > events:
> > > >> > > >> > > > > > - A user signs up to one of the user pool defined
> in
> > my
> > > >> cloud
> > > >> > > >> provider
> > > >> > > >> > > > > > - One of the databases used in my company has been
> > > >> updated
> > > >> > > >> > > > > > - A job in my cloud provider has been executed
> > > >> successfully
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > The intent of this AIP is to leverage datasets
> (which
> > > >> will be
> > > >> > > >> soon
> > > >> > > >> > > > > assets)
> > > >> > > >> > > > > > and update them based on external events. I would
> > like
> > > to
> > > >> > > >> propose this
> > > >> > > >> > > > > AIP
> > > >> > > >> > > > > > for discussion and more importantly, hear some
> > > feedbacks
> > > >> from
> > > >> > > >> you :)
> > > >> > > >> > > > > >
> > > >> > > >> > > > > >
> > > >> > > >> > > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > >> > > >> > > > <
> > > >> > > >> > > >
> > > >> > > >>
> > > >> > >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > >> > > >> > > > >
> > > >> > > >> > > > > >
> > > >> > > >> > > > > > Vincent
> > > >> > > >> > > > > >
> > > >> > > >> > > > >
> > > >> > > >> > > >
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >> >
> > > >>
> ---------------------------------------------------------------------
> > > >> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > > >> > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> > > >>
> > > >> > > >>
> > > >> > >
> > > >> >
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>
> > > >>
> > >
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
I was discussing this with Vincent. In either case, same as now or the one
proposed in the AIP, a user will have to use something like AWS
EventBridge[1] or GCS EventArc [2] where users will consume the event from
object storage (S3 object creation for example), and then they will have to
add the Airflow's Create Dataset endpoint to EventBridge [3]. Now, if you
just customize the payload to build the URI which is allowed (either via
GET / POST) in eventbridge, it works right now. However, with the current
proposal: a user will have to create a new user in Airflow and some mapping
to a function (that is either in the provider or a new user-defined
function) that can understand this specific payload, in this example the
payload for S3 events. This will become huge because this means that for
each payload, we will have to provide a new function and keep it updated.
From users POV, they will need to create a new user every time for a new
service (S3, Redshift, SNS, Bedrock etc). This will again likely have to go
to the Auth manager backend. Compared to what's available today -- i.e.
building a URI & extra metadata that can not only work with EventBridge or
Eventarc but by any service.
Since we already have to use things like EventBridge or EventArc for
managed service providers to transform the event, it fits well with the
existing approach. AWS Blog [3] even has a similar example for Datadog
where they use input transformer "{"detail":"$.detail"}" before sending it
to Datadog's API.
>"Having producer of an event generating even in their standard way, make
it easy for airflow to consume it as a dataset event without external
entities":
[1]: https://aws.amazon.com/eventbridge/ |
https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
[2]: https://cloud.google.com/eventarc/docs
[3]:
https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/
On Fri, 2 Aug 2024 at 00:14, Jarek Potiuk <ja...@potiuk.com> wrote:
> I proposed the mapping - because it's the easiest way (I think) to map
> between the "native" source to "airflow" target expectations. There are
> many producers of such events, and Airflow is the consumer. And it seems
> appropriate to have a way for our users to easily plug events produced from
> one system into our "events" API - without having to employ external
> "mapper" (say lambda) doing the conversion. While I think it is indeed "a
> bit odd", it's a solution that might leverage most of what we have -
> authorisation and API exposure via "user" in API.
>
> While I - myself - find it it a bit unusual, I think it might do the job,
> But I wonder if there is any alternative solution to the problem of "Having
> producer of an event generating even in their standard way, make it easy
> for airflow to consume it as a dataset event without external entities".
>
> On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > I would love for VOTE to get started on this one. I think most of the
> > commenters and those who replied to this email are happy with the
> proposal
> > on the poll-based approach.
> >
> > Regarding the push-based approach, I am not convinced that the proposed
> > implementation has any gains over what's already available with the
> Dataset
> > Event Create API; the one user-to-one function mapping is an odd user
> > experience. I'm curious to hear what others think.
> >
> > On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > I agree with both of you that it is indeed a good idea and that it can
> be
> > > added in Future work -- doesn't need to be part of this AIP.
> > >
> > > Thanks for the interest. I was not aware of such feature and this looks
> > >> really cool! I definitely think that can be useful for Airflow,
> > especially
> > >> for testing when you can easily replay events received in the past.
> > >> However, I do not think it should be part of the AIP and, as you
> > mentioned,
> > >> if should be a future work or a follow-up item of the AIP. Please let
> me
> > >> know if you (or anyone) disagree with this and we can talk about it.
> > >> Otherwise I'll update the future work section of the AIP and mention
> > this
> > >> archive and replay feature.
> > >
> > >
> > > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org> wrote:
> > >
> > >> Hey Pavan,
> > >>
> > >> Thanks for the interest. I was not aware of such feature and this
> looks
> > >> really cool! I definitely think that can be useful for Airflow,
> > especially
> > >> for testing when you can easily replay events received in the past.
> > >> However, I do not think it should be part of the AIP and, as you
> > mentioned,
> > >> if should be a future work or a follow-up item of the AIP. Please let
> me
> > >> know if you (or anyone) disagree with this and we can talk about it.
> > >> Otherwise I'll update the future work section of the AIP and mention
> > this
> > >> archive and replay feature.
> > >>
> > >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > >> > Thanks Vincent, I took a look , this is really good. Don't have
> access
> > >> to
> > >> > the confluence page to comment :) so adding it here.
> > >> >
> > >> > As events arrive-->do somework-->end.
> > >> >
> > >> > So I'm uncertain if my comment pertains to the current poll/push
> model
> > >> or
> > >> > if it fits part of future work(seen event batching ).
> > >> >
> > >> > Have you given any thought to the event archival mechanism and event
> > >> > replay? This could significantly aid in testing and recovery of
> > workflow
> > >> > and testing new functionality with events by just replay the events.
> > The
> > >> > archival mechanism I am referring to is similar to today in AWS we
> > have
> > >> > Event Bridge Archive and Replay.
> > >> >
> > >> > Regards,
> > >> > Pavan
> > >> >
> > >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > >> >
> > >> > > I actually did manage to take a look, thanks for the work. I am +1
> > on
> > >> the
> > >> > > poll-based approach -- left a comment on the push-based: I am not
> > >> sure of
> > >> > > why we need a function since create asset event API endpoint
> should
> > >> have
> > >> > > all info needed for what the Asset was.
> > >> > >
> > >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com>
> > wrote:
> > >> > >
> > >> > > > Thanks Vincent, I will take a look again tomorrow.
> > >> > > >
> > >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vincbeck@apache.org
> >
> > >> wrote:
> > >> > > >
> > >> > > >> Hi everyone,
> > >> > > >>
> > >> > > >> I updated the AIP-82 given the different comments and concerns
> I
> > >> > > >> received. I also tried to reply to all comments individually. I
> > >> would
> > >> > > >> really appreciate if you can do a second pass and let me know
> > what
> > >> you
> > >> > > >> think. Overall, this is what I changed in the AIP:
> > >> > > >>
> > >> > > >> - Push based event-driven scheduling. I updated this section
> > >> entirely
> > >> > > >> because I received many concerns about the previous proposal.
> The
> > >> > > overall
> > >> > > >> idea now is to leverage the create asset event API endpoint to
> > send
> > >> > > >> notifications from external (e.g. cloud provider) to Airflow
> > >> > > environment.
> > >> > > >>
> > >> > > >> - I updated the poll based event-driven scheduling DAG author
> > >> experience
> > >> > > >> to use a message queue scenario. I understood that this is
> > >> probably the
> > >> > > >> main use case we are trying to cover with this AIP, thus I used
> > it
> > >> as
> > >> > > >> example and mentioned it multiple times across the AIP.
> > >> > > >>
> > >> > > >> Thanks again for your time :)
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > >> > > >>
> > >> > > >> Vincent
> > >> > > >>
> > >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > >> > > >> > Thanks a lot all for the comments, this is very much
> > >> appreciated! I
> > >> > > >> received many comments from this thread and in confluence,
> thanks
> > >> again.
> > >> > > >> I'll try to address them all in the AIP and will send an email
> in
> > >> this
> > >> > > >> thread once done. I will most likely revisit the push-based
> > >> approach
> > >> > > given
> > >> > > >> the number of concerns I received, thanks Jarek for proposing
> > >> another
> > >> > > >> solution, I'll probably go down that path.
> > >> > > >> >
> > >> > > >> > One follow-up question Vikram.
> > >> > > >> >
> > >> > > >> > > The bespoke triggerer approach completely makes sense for
> the
> > >> long
> > >> > > >> tail here, but can we do better for the 20% of scenarios which
> > >> cover
> > >> > > well
> > >> > > >> over 80% of usage here is the question in my mind. Or, are you
> > >> thinking
> > >> > > of
> > >> > > >> those as being covered in the "push" model?
> > >> > > >> >
> > >> > > >> > Could you share more details about what is this "20% of
> > scenarios
> > >> > > which
> > >> > > >> cover well over 80% of usage" please?
> > >> > > >> >
> > >> > > >> > Vincent
> > >> > > >> >
> > >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > >> > > >> > > Thanks Vincent for driving these, I have added my comments
> to
> > >> the
> > >> > > AIP
> > >> > > >> too.
> > >> > > >> > >
> > >> > > >> > > Regards,
> > >> > > >> > > Kaxil
> > >> > > >> > >
> > >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> > (XC-AS/EAE-ADA-T)
> > >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > >> > > >> > >
> > >> > > >> > > > +1 on the comments of Vikram and Jarek, added main points
> > on
> > >> > > >> confluence
> > >> > > >> > > >
> > >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > >> > > >> > > > ________________________________
> > >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > >> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > >> > > >> > > > Subject: Re: [DISCUSS] External event driven scheduling
> in
> > >> Airflow
> > >> > > >> > > >
> > >> > > >> > > > Vincent,
> > >> > > >> > > >
> > >> > > >> > > > Thanks for writing this up. The overview looks really
> good!
> > >> > > >> > > >
> > >> > > >> > > > I will leave my comments in the AIP as well, but at a
> high
> > >> level
> > >> > > >> they are
> > >> > > >> > > > both relatively focused on the "how", rather than the
> > "what".
> > >> > > >> > > > With respect to the pull / polling approach, I completely
> > >> agree
> > >> > > >> that some
> > >> > > >> > > > incarnation of this is needed.
> > >> > > >> > > > I am less certain as to how on this part. The bespoke
> > >> triggerer
> > >> > > >> approach
> > >> > > >> > > > completely makes sense for the long tail here, but can we
> > do
> > >> > > better
> > >> > > >> for the
> > >> > > >> > > > 20% of scenarios which cover well over 80% of usage here
> is
> > >> the
> > >> > > >> question in
> > >> > > >> > > > my mind. Or, are you thinking of those as being covered
> in
> > >> the
> > >> > > >> "push"
> > >> > > >> > > > model?
> > >> > > >> > > >
> > >> > > >> > > > Which leads to the "push" model approach.
> > >> > > >> > > > I am struggling with the same question that Jarek raised
> > here
> > >> > > about
> > >> > > >> whether
> > >> > > >> > > > we need a new Airflow entity over and beyond the existing
> > >> REST API
> > >> > > >> for the
> > >> > > >> > > > same.
> > >> > > >> > > > I am concerned about this becoming a vector of attack on
> > >> Airflow.
> > >> > > >> > > > I see that this is a hot topic of discussion in the
> > >> Confluence doc
> > >> > > >> as well,
> > >> > > >> > > > but wanted to summarize here as well, so it didn't get
> lost
> > >> in the
> > >> > > >> threads
> > >> > > >> > > > of comments.
> > >> > > >> > > >
> > >> > > >> > > > Best regards,
> > >> > > >> > > > Vikram
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> > >> jarek@potiuk.com>
> > >> > > >> wrote:
> > >> > > >> > > >
> > >> > > >> > > > > Thanks Vincent. I took a look and I have a general
> > >> comment. I
> > >> > > >> > > > > strongly think external driven scheduling is really
> > needed
> > >> -
> > >> > > >> especially,
> > >> > > >> > > > it
> > >> > > >> > > > > should be much easier for a user to "plug-in" such an
> > >> external
> > >> > > >> event to
> > >> > > >> > > > > Airflow. And there are two parts of it - as correctly
> > >> stated
> > >> > > >> there - pull
> > >> > > >> > > > > and push.
> > >> > > >> > > > >
> > >> > > >> > > > > For the pull - I think it would be great to have a kind
> > of
> > >> > > >> specialized
> > >> > > >> > > > > Triggers that will be started when DAG is parsed - and
> > >> those
> > >> > > >> Triggers
> > >> > > >> > > > could
> > >> > > >> > > > > generate the events for DAGs. I think basically that's
> > all
> > >> that
> > >> > > is
> > >> > > >> > > > needed,
> > >> > > >> > > > > for example I imagine a pubsub trigger that will
> > subscribe
> > >> to
> > >> > > >> messages
> > >> > > >> > > > > coming on the pubsub queue and fire "Asset" event when
> a
> > >> message
> > >> > > >> is
> > >> > > >> > > > > received. Not much controversy there - I am not sure
> > about
> > >> the
> > >> > > >> polling
> > >> > > >> > > > > thing , because I've always believed that when
> > >> "asyncio-native"
> > >> > > >> Trigger
> > >> > > >> > > > is
> > >> > > >> > > > > run in the asyncio event loop, we do not "poll" every
> > >> second or
> > >> > > >> so (but
> > >> > > >> > > > > maybe this is just coming from some specific triggers
> > that
> > >> > > >> actually do
> > >> > > >> > > > > such regular poll. But yes - there are polls like
> > running
> > >> > > select
> > >> > > >> on the
> > >> > > >> > > > DB
> > >> > > >> > > > > that cannot be easily "async-ed" so having a
> configurable
> > >> > > polling
> > >> > > >> time
> > >> > > >> > > > > would be good there (but I am not sure maybe it's even
> > >> possible
> > >> > > >> today). I
> > >> > > >> > > > > think this would be really great if we have that
> option,
> > >> because
> > >> > > >> it makes
> > >> > > >> > > > > it much easier to set up the authorization for Airlfow
> > >> users -
> > >> > > >> rather
> > >> > > >> > > > than
> > >> > > >> > > > > setting up authorization and REST calls coming from an
> > >> external
> > >> > > >> system,
> > >> > > >> > > > we
> > >> > > >> > > > > can utilize Connections of Airlfow to authorize such a
> > >> Trigger
> > >> > > to
> > >> > > >> > > > subscribe
> > >> > > >> > > > > to events.
> > >> > > >> > > > >
> > >> > > >> > > > > For the push proposal - as I read the proposal, the
> main
> > >> point
> > >> > > >> behind it
> > >> > > >> > > > > is rather than users having to write "Airflow" way of
> > >> triggering
> > >> > > >> events
> > >> > > >> > > > and
> > >> > > >> > > > > configuring authentication (using REST API) to generate
> > >> asset
> > >> > > >> events, is
> > >> > > >> > > > to
> > >> > > >> > > > > make Airflow natively understand external ways of
> pushing
> > >> - and
> > >> > > >> > > > effectively
> > >> > > >> > > > > authorizing and mapping such incoming unauthorized
> > >> requests into
> > >> > > >> event
> > >> > > >> > > > that
> > >> > > >> > > > > could be generated by an API REST call.
> > >> > > >> > > > > I am not really sure honestly if this is something that
> > we
> > >> want
> > >> > > as
> > >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such an
> > >> > > unauthorised
> > >> > > >> > > > endpoint
> > >> > > >> > > > > is probably not a good idea - for a variety of reasons,
> > >> mostly
> > >> > > >> security.
> > >> > > >> > > > > And as I understand the goal is that users can easily
> > >> point at
> > >> > > >> > > > "3rd-party"
> > >> > > >> > > > > notification to Airflow and get the event generated.
> > >> > > >> > > > >
> > >> > > >> > > > > My feeling is that while this is needed - it should be
> > >> > > >> externalised from
> > >> > > >> > > > > airlfow webserver. The authorization has to be set up
> > >> anyway
> > >> > > >> > > > additionally -
> > >> > > >> > > > > unlike in "poll" case - we cannot use Connections for
> > >> > > authorizing
> > >> > > >> > > > (because
> > >> > > >> > > > > it's not Airlfow that authorizes in an external system
> -
> > >> it's
> > >> > > the
> > >> > > >> other
> > >> > > >> > > > way
> > >> > > >> > > > > round). So we have to anyhow setup "something extra" in
> > >> Airflow
> > >> > > to
> > >> > > >> > > > > authorize the external system. Which could be what we
> > have
> > >> now -
> > >> > > >> user
> > >> > > >> > > > that
> > >> > > >> > > > > allows us to trigger the event. Which means that our
> REST
> > >> API
> > >> > > >> could
> > >> > > >> > > > > potentially be used the same way it is now, but we will
> > >> need
> > >> > > >> "something"
> > >> > > >> > > > > (library, lambda function etc.) that users could easily
> > >> setup in
> > >> > > >> the
> > >> > > >> > > > > external system to map whatever trigger they generate
> > >> natively
> > >> > > >> (say S3
> > >> > > >> > > > file
> > >> > > >> > > > > created) to Airflow REST API.
> > >> > > >> > > > >
> > >> > > >> > > > > As I see it - this is quite often used (and very
> > >> practical, that
> > >> > > >> you
> > >> > > >> > > > deploy
> > >> > > >> > > > > a cloud function or lambda that subscribes on the event
> > >> received
> > >> > > >> when
> > >> > > >> > > > > S3/GCS is created. So it would be on the user to deploy
> > >> such a
> > >> > > >> lambda -
> > >> > > >> > > > but
> > >> > > >> > > > > we **could** provide a library of those: say s3 lambda,
> > gcp
> > >> > > cloud
> > >> > > >> > > > function
> > >> > > >> > > > > in respective providers - with documentation how to set
> > >> them up,
> > >> > > >> and how
> > >> > > >> > > > to
> > >> > > >> > > > > configure authorization and we would be generally
> "done".
> > >> I am
> > >> > > >> just not
> > >> > > >> > > > > sure if we need a new entity in Airflow for that (Event
> > >> > > >> receiver). It
> > >> > > >> > > > feels
> > >> > > >> > > > > like it asks Airflow to take more responsibility, when
> we
> > >> all
> > >> > > >> think on
> > >> > > >> > > > what
> > >> > > >> > > > > to "remove" from Airflow rather than "add" to it -
> > >> especially
> > >> > > >> when it
> > >> > > >> > > > comes
> > >> > > >> > > > > to external integrations. It feels to me that Airflow
> > >> should
> > >> > > make
> > >> > > >> it easy
> > >> > > >> > > > > to be triggered by such an external system and make it
> > >> easy to
> > >> > > >> "map" to
> > >> > > >> > > > the
> > >> > > >> > > > > way we expect to get events triggered, but this should
> be
> > >> done
> > >> > > >> outside of
> > >> > > >> > > > > Airflow. If the users can easily find in our docs when
> > they
> > >> > > >> search "what
> > >> > > >> > > > do
> > >> > > >> > > > > I do to externally trigger Airflow on S3 change":
> either
> > a)
> > >> > > >> configure
> > >> > > >> > > > > polling in airflow using s3 Connection, or b) "create a
> > >> user +
> > >> > > >> deploy
> > >> > > >> > > > this
> > >> > > >> > > > > lambda with those parameters" - that is "easy enough"
> > and
> > >> very
> > >> > > >> practical
> > >> > > >> > > > > as well.
> > >> > > >> > > > >
> > >> > > >> > > > > But maybe I am not seeing the whole picture and the
> real
> > >> problem
> > >> > > >> it's
> > >> > > >> > > > > solving - so take it as a "first review pass" and "guts
> > >> > > feeling".
> > >> > > >> > > > >
> > >> > > >> > > > > J.
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > >> > > >> > > > <vincbeck@amazon.com.invalid
> > >> > > >> > > > > >
> > >> > > >> > > > > wrote:
> > >> > > >> > > > >
> > >> > > >> > > > > > Hello everyone,
> > >> > > >> > > > > >
> > >> > > >> > > > > > I created a draft AIP regarding "External event
> driven
> > >> > > >> scheduling in
> > >> > > >> > > > > > Airflow". This proposal is about adding capability in
> > >> Airflow
> > >> > > to
> > >> > > >> > > > schedule
> > >> > > >> > > > > > DAGs based on external events. Here are some examples
> > of
> > >> such
> > >> > > >> external
> > >> > > >> > > > > > events:
> > >> > > >> > > > > > - A user signs up to one of the user pool defined in
> my
> > >> cloud
> > >> > > >> provider
> > >> > > >> > > > > > - One of the databases used in my company has been
> > >> updated
> > >> > > >> > > > > > - A job in my cloud provider has been executed
> > >> successfully
> > >> > > >> > > > > >
> > >> > > >> > > > > > The intent of this AIP is to leverage datasets (which
> > >> will be
> > >> > > >> soon
> > >> > > >> > > > > assets)
> > >> > > >> > > > > > and update them based on external events. I would
> like
> > to
> > >> > > >> propose this
> > >> > > >> > > > > AIP
> > >> > > >> > > > > > for discussion and more importantly, hear some
> > feedbacks
> > >> from
> > >> > > >> you :)
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >>
> > >> > >
> > >>
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > >> > > >> > > > <
> > >> > > >> > > >
> > >> > > >>
> > >> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > >> > > >> > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > > > Vincent
> > >> > > >> > > > > >
> > >> > > >> > > > >
> > >> > > >> > > >
> > >> > > >> > >
> > >> > > >> >
> > >> > > >> >
> > >> ---------------------------------------------------------------------
> > >> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > > >> >
> > >> > > >> >
> > >> > > >>
> > >> > > >>
> > >> ---------------------------------------------------------------------
> > >> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > > >>
> > >> > > >>
> > >> > >
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>
> > >>
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
I proposed the mapping - because it's the easiest way (I think) to map
between the "native" source to "airflow" target expectations. There are
many producers of such events, and Airflow is the consumer. And it seems
appropriate to have a way for our users to easily plug events produced from
one system into our "events" API - without having to employ external
"mapper" (say lambda) doing the conversion. While I think it is indeed "a
bit odd", it's a solution that might leverage most of what we have -
authorisation and API exposure via "user" in API.
While I - myself - find it it a bit unusual, I think it might do the job,
But I wonder if there is any alternative solution to the problem of "Having
producer of an event generating even in their standard way, make it easy
for airflow to consume it as a dataset event without external entities".
On Fri, Aug 2, 2024 at 12:57 AM Kaxil Naik <ka...@gmail.com> wrote:
> I would love for VOTE to get started on this one. I think most of the
> commenters and those who replied to this email are happy with the proposal
> on the poll-based approach.
>
> Regarding the push-based approach, I am not convinced that the proposed
> implementation has any gains over what's already available with the Dataset
> Event Create API; the one user-to-one function mapping is an odd user
> experience. I'm curious to hear what others think.
>
> On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com> wrote:
>
> > I agree with both of you that it is indeed a good idea and that it can be
> > added in Future work -- doesn't need to be part of this AIP.
> >
> > Thanks for the interest. I was not aware of such feature and this looks
> >> really cool! I definitely think that can be useful for Airflow,
> especially
> >> for testing when you can easily replay events received in the past.
> >> However, I do not think it should be part of the AIP and, as you
> mentioned,
> >> if should be a future work or a follow-up item of the AIP. Please let me
> >> know if you (or anyone) disagree with this and we can talk about it.
> >> Otherwise I'll update the future work section of the AIP and mention
> this
> >> archive and replay feature.
> >
> >
> > On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org> wrote:
> >
> >> Hey Pavan,
> >>
> >> Thanks for the interest. I was not aware of such feature and this looks
> >> really cool! I definitely think that can be useful for Airflow,
> especially
> >> for testing when you can easily replay events received in the past.
> >> However, I do not think it should be part of the AIP and, as you
> mentioned,
> >> if should be a future work or a follow-up item of the AIP. Please let me
> >> know if you (or anyone) disagree with this and we can talk about it.
> >> Otherwise I'll update the future work section of the AIP and mention
> this
> >> archive and replay feature.
> >>
> >> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> >> > Thanks Vincent, I took a look , this is really good. Don't have access
> >> to
> >> > the confluence page to comment :) so adding it here.
> >> >
> >> > As events arrive-->do somework-->end.
> >> >
> >> > So I'm uncertain if my comment pertains to the current poll/push model
> >> or
> >> > if it fits part of future work(seen event batching ).
> >> >
> >> > Have you given any thought to the event archival mechanism and event
> >> > replay? This could significantly aid in testing and recovery of
> workflow
> >> > and testing new functionality with events by just replay the events.
> The
> >> > archival mechanism I am referring to is similar to today in AWS we
> have
> >> > Event Bridge Archive and Replay.
> >> >
> >> > Regards,
> >> > Pavan
> >> >
> >> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com>
> wrote:
> >> >
> >> > > I actually did manage to take a look, thanks for the work. I am +1
> on
> >> the
> >> > > poll-based approach -- left a comment on the push-based: I am not
> >> sure of
> >> > > why we need a function since create asset event API endpoint should
> >> have
> >> > > all info needed for what the Asset was.
> >> > >
> >> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com>
> wrote:
> >> > >
> >> > > > Thanks Vincent, I will take a look again tomorrow.
> >> > > >
> >> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org>
> >> wrote:
> >> > > >
> >> > > >> Hi everyone,
> >> > > >>
> >> > > >> I updated the AIP-82 given the different comments and concerns I
> >> > > >> received. I also tried to reply to all comments individually. I
> >> would
> >> > > >> really appreciate if you can do a second pass and let me know
> what
> >> you
> >> > > >> think. Overall, this is what I changed in the AIP:
> >> > > >>
> >> > > >> - Push based event-driven scheduling. I updated this section
> >> entirely
> >> > > >> because I received many concerns about the previous proposal. The
> >> > > overall
> >> > > >> idea now is to leverage the create asset event API endpoint to
> send
> >> > > >> notifications from external (e.g. cloud provider) to Airflow
> >> > > environment.
> >> > > >>
> >> > > >> - I updated the poll based event-driven scheduling DAG author
> >> experience
> >> > > >> to use a message queue scenario. I understood that this is
> >> probably the
> >> > > >> main use case we are trying to cover with this AIP, thus I used
> it
> >> as
> >> > > >> example and mentioned it multiple times across the AIP.
> >> > > >>
> >> > > >> Thanks again for your time :)
> >> > > >>
> >> > > >>
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> >> > > >>
> >> > > >> Vincent
> >> > > >>
> >> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> >> > > >> > Thanks a lot all for the comments, this is very much
> >> appreciated! I
> >> > > >> received many comments from this thread and in confluence, thanks
> >> again.
> >> > > >> I'll try to address them all in the AIP and will send an email in
> >> this
> >> > > >> thread once done. I will most likely revisit the push-based
> >> approach
> >> > > given
> >> > > >> the number of concerns I received, thanks Jarek for proposing
> >> another
> >> > > >> solution, I'll probably go down that path.
> >> > > >> >
> >> > > >> > One follow-up question Vikram.
> >> > > >> >
> >> > > >> > > The bespoke triggerer approach completely makes sense for the
> >> long
> >> > > >> tail here, but can we do better for the 20% of scenarios which
> >> cover
> >> > > well
> >> > > >> over 80% of usage here is the question in my mind. Or, are you
> >> thinking
> >> > > of
> >> > > >> those as being covered in the "push" model?
> >> > > >> >
> >> > > >> > Could you share more details about what is this "20% of
> scenarios
> >> > > which
> >> > > >> cover well over 80% of usage" please?
> >> > > >> >
> >> > > >> > Vincent
> >> > > >> >
> >> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> >> > > >> > > Thanks Vincent for driving these, I have added my comments to
> >> the
> >> > > AIP
> >> > > >> too.
> >> > > >> > >
> >> > > >> > > Regards,
> >> > > >> > > Kaxil
> >> > > >> > >
> >> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens
> (XC-AS/EAE-ADA-T)
> >> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> >> > > >> > >
> >> > > >> > > > +1 on the comments of Vikram and Jarek, added main points
> on
> >> > > >> confluence
> >> > > >> > > >
> >> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> >> > > >> > > > ________________________________
> >> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> >> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> >> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> >> > > >> > > > Subject: Re: [DISCUSS] External event driven scheduling in
> >> Airflow
> >> > > >> > > >
> >> > > >> > > > Vincent,
> >> > > >> > > >
> >> > > >> > > > Thanks for writing this up. The overview looks really good!
> >> > > >> > > >
> >> > > >> > > > I will leave my comments in the AIP as well, but at a high
> >> level
> >> > > >> they are
> >> > > >> > > > both relatively focused on the "how", rather than the
> "what".
> >> > > >> > > > With respect to the pull / polling approach, I completely
> >> agree
> >> > > >> that some
> >> > > >> > > > incarnation of this is needed.
> >> > > >> > > > I am less certain as to how on this part. The bespoke
> >> triggerer
> >> > > >> approach
> >> > > >> > > > completely makes sense for the long tail here, but can we
> do
> >> > > better
> >> > > >> for the
> >> > > >> > > > 20% of scenarios which cover well over 80% of usage here is
> >> the
> >> > > >> question in
> >> > > >> > > > my mind. Or, are you thinking of those as being covered in
> >> the
> >> > > >> "push"
> >> > > >> > > > model?
> >> > > >> > > >
> >> > > >> > > > Which leads to the "push" model approach.
> >> > > >> > > > I am struggling with the same question that Jarek raised
> here
> >> > > about
> >> > > >> whether
> >> > > >> > > > we need a new Airflow entity over and beyond the existing
> >> REST API
> >> > > >> for the
> >> > > >> > > > same.
> >> > > >> > > > I am concerned about this becoming a vector of attack on
> >> Airflow.
> >> > > >> > > > I see that this is a hot topic of discussion in the
> >> Confluence doc
> >> > > >> as well,
> >> > > >> > > > but wanted to summarize here as well, so it didn't get lost
> >> in the
> >> > > >> threads
> >> > > >> > > > of comments.
> >> > > >> > > >
> >> > > >> > > > Best regards,
> >> > > >> > > > Vikram
> >> > > >> > > >
> >> > > >> > > >
> >> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> >> jarek@potiuk.com>
> >> > > >> wrote:
> >> > > >> > > >
> >> > > >> > > > > Thanks Vincent. I took a look and I have a general
> >> comment. I
> >> > > >> > > > > strongly think external driven scheduling is really
> needed
> >> -
> >> > > >> especially,
> >> > > >> > > > it
> >> > > >> > > > > should be much easier for a user to "plug-in" such an
> >> external
> >> > > >> event to
> >> > > >> > > > > Airflow. And there are two parts of it - as correctly
> >> stated
> >> > > >> there - pull
> >> > > >> > > > > and push.
> >> > > >> > > > >
> >> > > >> > > > > For the pull - I think it would be great to have a kind
> of
> >> > > >> specialized
> >> > > >> > > > > Triggers that will be started when DAG is parsed - and
> >> those
> >> > > >> Triggers
> >> > > >> > > > could
> >> > > >> > > > > generate the events for DAGs. I think basically that's
> all
> >> that
> >> > > is
> >> > > >> > > > needed,
> >> > > >> > > > > for example I imagine a pubsub trigger that will
> subscribe
> >> to
> >> > > >> messages
> >> > > >> > > > > coming on the pubsub queue and fire "Asset" event when a
> >> message
> >> > > >> is
> >> > > >> > > > > received. Not much controversy there - I am not sure
> about
> >> the
> >> > > >> polling
> >> > > >> > > > > thing , because I've always believed that when
> >> "asyncio-native"
> >> > > >> Trigger
> >> > > >> > > > is
> >> > > >> > > > > run in the asyncio event loop, we do not "poll" every
> >> second or
> >> > > >> so (but
> >> > > >> > > > > maybe this is just coming from some specific triggers
> that
> >> > > >> actually do
> >> > > >> > > > > such regular poll. But yes - there are polls like
> running
> >> > > select
> >> > > >> on the
> >> > > >> > > > DB
> >> > > >> > > > > that cannot be easily "async-ed" so having a configurable
> >> > > polling
> >> > > >> time
> >> > > >> > > > > would be good there (but I am not sure maybe it's even
> >> possible
> >> > > >> today). I
> >> > > >> > > > > think this would be really great if we have that option,
> >> because
> >> > > >> it makes
> >> > > >> > > > > it much easier to set up the authorization for Airlfow
> >> users -
> >> > > >> rather
> >> > > >> > > > than
> >> > > >> > > > > setting up authorization and REST calls coming from an
> >> external
> >> > > >> system,
> >> > > >> > > > we
> >> > > >> > > > > can utilize Connections of Airlfow to authorize such a
> >> Trigger
> >> > > to
> >> > > >> > > > subscribe
> >> > > >> > > > > to events.
> >> > > >> > > > >
> >> > > >> > > > > For the push proposal - as I read the proposal, the main
> >> point
> >> > > >> behind it
> >> > > >> > > > > is rather than users having to write "Airflow" way of
> >> triggering
> >> > > >> events
> >> > > >> > > > and
> >> > > >> > > > > configuring authentication (using REST API) to generate
> >> asset
> >> > > >> events, is
> >> > > >> > > > to
> >> > > >> > > > > make Airflow natively understand external ways of pushing
> >> - and
> >> > > >> > > > effectively
> >> > > >> > > > > authorizing and mapping such incoming unauthorized
> >> requests into
> >> > > >> event
> >> > > >> > > > that
> >> > > >> > > > > could be generated by an API REST call.
> >> > > >> > > > > I am not really sure honestly if this is something that
> we
> >> want
> >> > > as
> >> > > >> > > > > "running" in airlfow as an endpoint. I'd say such an
> >> > > unauthorised
> >> > > >> > > > endpoint
> >> > > >> > > > > is probably not a good idea - for a variety of reasons,
> >> mostly
> >> > > >> security.
> >> > > >> > > > > And as I understand the goal is that users can easily
> >> point at
> >> > > >> > > > "3rd-party"
> >> > > >> > > > > notification to Airflow and get the event generated.
> >> > > >> > > > >
> >> > > >> > > > > My feeling is that while this is needed - it should be
> >> > > >> externalised from
> >> > > >> > > > > airlfow webserver. The authorization has to be set up
> >> anyway
> >> > > >> > > > additionally -
> >> > > >> > > > > unlike in "poll" case - we cannot use Connections for
> >> > > authorizing
> >> > > >> > > > (because
> >> > > >> > > > > it's not Airlfow that authorizes in an external system -
> >> it's
> >> > > the
> >> > > >> other
> >> > > >> > > > way
> >> > > >> > > > > round). So we have to anyhow setup "something extra" in
> >> Airflow
> >> > > to
> >> > > >> > > > > authorize the external system. Which could be what we
> have
> >> now -
> >> > > >> user
> >> > > >> > > > that
> >> > > >> > > > > allows us to trigger the event. Which means that our REST
> >> API
> >> > > >> could
> >> > > >> > > > > potentially be used the same way it is now, but we will
> >> need
> >> > > >> "something"
> >> > > >> > > > > (library, lambda function etc.) that users could easily
> >> setup in
> >> > > >> the
> >> > > >> > > > > external system to map whatever trigger they generate
> >> natively
> >> > > >> (say S3
> >> > > >> > > > file
> >> > > >> > > > > created) to Airflow REST API.
> >> > > >> > > > >
> >> > > >> > > > > As I see it - this is quite often used (and very
> >> practical, that
> >> > > >> you
> >> > > >> > > > deploy
> >> > > >> > > > > a cloud function or lambda that subscribes on the event
> >> received
> >> > > >> when
> >> > > >> > > > > S3/GCS is created. So it would be on the user to deploy
> >> such a
> >> > > >> lambda -
> >> > > >> > > > but
> >> > > >> > > > > we **could** provide a library of those: say s3 lambda,
> gcp
> >> > > cloud
> >> > > >> > > > function
> >> > > >> > > > > in respective providers - with documentation how to set
> >> them up,
> >> > > >> and how
> >> > > >> > > > to
> >> > > >> > > > > configure authorization and we would be generally "done".
> >> I am
> >> > > >> just not
> >> > > >> > > > > sure if we need a new entity in Airflow for that (Event
> >> > > >> receiver). It
> >> > > >> > > > feels
> >> > > >> > > > > like it asks Airflow to take more responsibility, when we
> >> all
> >> > > >> think on
> >> > > >> > > > what
> >> > > >> > > > > to "remove" from Airflow rather than "add" to it -
> >> especially
> >> > > >> when it
> >> > > >> > > > comes
> >> > > >> > > > > to external integrations. It feels to me that Airflow
> >> should
> >> > > make
> >> > > >> it easy
> >> > > >> > > > > to be triggered by such an external system and make it
> >> easy to
> >> > > >> "map" to
> >> > > >> > > > the
> >> > > >> > > > > way we expect to get events triggered, but this should be
> >> done
> >> > > >> outside of
> >> > > >> > > > > Airflow. If the users can easily find in our docs when
> they
> >> > > >> search "what
> >> > > >> > > > do
> >> > > >> > > > > I do to externally trigger Airflow on S3 change": either
> a)
> >> > > >> configure
> >> > > >> > > > > polling in airflow using s3 Connection, or b) "create a
> >> user +
> >> > > >> deploy
> >> > > >> > > > this
> >> > > >> > > > > lambda with those parameters" - that is "easy enough"
> and
> >> very
> >> > > >> practical
> >> > > >> > > > > as well.
> >> > > >> > > > >
> >> > > >> > > > > But maybe I am not seeing the whole picture and the real
> >> problem
> >> > > >> it's
> >> > > >> > > > > solving - so take it as a "first review pass" and "guts
> >> > > feeling".
> >> > > >> > > > >
> >> > > >> > > > > J.
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > > >
> >> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> >> > > >> > > > <vincbeck@amazon.com.invalid
> >> > > >> > > > > >
> >> > > >> > > > > wrote:
> >> > > >> > > > >
> >> > > >> > > > > > Hello everyone,
> >> > > >> > > > > >
> >> > > >> > > > > > I created a draft AIP regarding "External event driven
> >> > > >> scheduling in
> >> > > >> > > > > > Airflow". This proposal is about adding capability in
> >> Airflow
> >> > > to
> >> > > >> > > > schedule
> >> > > >> > > > > > DAGs based on external events. Here are some examples
> of
> >> such
> >> > > >> external
> >> > > >> > > > > > events:
> >> > > >> > > > > > - A user signs up to one of the user pool defined in my
> >> cloud
> >> > > >> provider
> >> > > >> > > > > > - One of the databases used in my company has been
> >> updated
> >> > > >> > > > > > - A job in my cloud provider has been executed
> >> successfully
> >> > > >> > > > > >
> >> > > >> > > > > > The intent of this AIP is to leverage datasets (which
> >> will be
> >> > > >> soon
> >> > > >> > > > > assets)
> >> > > >> > > > > > and update them based on external events. I would like
> to
> >> > > >> propose this
> >> > > >> > > > > AIP
> >> > > >> > > > > > for discussion and more importantly, hear some
> feedbacks
> >> from
> >> > > >> you :)
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >>
> >> > >
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> >> > > >> > > > <
> >> > > >> > > >
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> >> > > >> > > > >
> >> > > >> > > > > >
> >> > > >> > > > > > Vincent
> >> > > >> > > > > >
> >> > > >> > > > >
> >> > > >> > > >
> >> > > >> > >
> >> > > >> >
> >> > > >> >
> >> ---------------------------------------------------------------------
> >> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> >> > > >> >
> >> > > >> >
> >> > > >>
> >> > > >>
> >> ---------------------------------------------------------------------
> >> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> >> > > >>
> >> > > >>
> >> > >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>
> >>
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
I would love for VOTE to get started on this one. I think most of the
commenters and those who replied to this email are happy with the proposal
on the poll-based approach.
Regarding the push-based approach, I am not convinced that the proposed
implementation has any gains over what's already available with the Dataset
Event Create API; the one user-to-one function mapping is an odd user
experience. I'm curious to hear what others think.
On Thu, 1 Aug 2024 at 17:39, Kaxil Naik <ka...@gmail.com> wrote:
> I agree with both of you that it is indeed a good idea and that it can be
> added in Future work -- doesn't need to be part of this AIP.
>
> Thanks for the interest. I was not aware of such feature and this looks
>> really cool! I definitely think that can be useful for Airflow, especially
>> for testing when you can easily replay events received in the past.
>> However, I do not think it should be part of the AIP and, as you mentioned,
>> if should be a future work or a follow-up item of the AIP. Please let me
>> know if you (or anyone) disagree with this and we can talk about it.
>> Otherwise I'll update the future work section of the AIP and mention this
>> archive and replay feature.
>
>
> On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org> wrote:
>
>> Hey Pavan,
>>
>> Thanks for the interest. I was not aware of such feature and this looks
>> really cool! I definitely think that can be useful for Airflow, especially
>> for testing when you can easily replay events received in the past.
>> However, I do not think it should be part of the AIP and, as you mentioned,
>> if should be a future work or a follow-up item of the AIP. Please let me
>> know if you (or anyone) disagree with this and we can talk about it.
>> Otherwise I'll update the future work section of the AIP and mention this
>> archive and replay feature.
>>
>> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
>> > Thanks Vincent, I took a look , this is really good. Don't have access
>> to
>> > the confluence page to comment :) so adding it here.
>> >
>> > As events arrive-->do somework-->end.
>> >
>> > So I'm uncertain if my comment pertains to the current poll/push model
>> or
>> > if it fits part of future work(seen event batching ).
>> >
>> > Have you given any thought to the event archival mechanism and event
>> > replay? This could significantly aid in testing and recovery of workflow
>> > and testing new functionality with events by just replay the events. The
>> > archival mechanism I am referring to is similar to today in AWS we have
>> > Event Bridge Archive and Replay.
>> >
>> > Regards,
>> > Pavan
>> >
>> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com> wrote:
>> >
>> > > I actually did manage to take a look, thanks for the work. I am +1 on
>> the
>> > > poll-based approach -- left a comment on the push-based: I am not
>> sure of
>> > > why we need a function since create asset event API endpoint should
>> have
>> > > all info needed for what the Asset was.
>> > >
>> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com> wrote:
>> > >
>> > > > Thanks Vincent, I will take a look again tomorrow.
>> > > >
>> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org>
>> wrote:
>> > > >
>> > > >> Hi everyone,
>> > > >>
>> > > >> I updated the AIP-82 given the different comments and concerns I
>> > > >> received. I also tried to reply to all comments individually. I
>> would
>> > > >> really appreciate if you can do a second pass and let me know what
>> you
>> > > >> think. Overall, this is what I changed in the AIP:
>> > > >>
>> > > >> - Push based event-driven scheduling. I updated this section
>> entirely
>> > > >> because I received many concerns about the previous proposal. The
>> > > overall
>> > > >> idea now is to leverage the create asset event API endpoint to send
>> > > >> notifications from external (e.g. cloud provider) to Airflow
>> > > environment.
>> > > >>
>> > > >> - I updated the poll based event-driven scheduling DAG author
>> experience
>> > > >> to use a message queue scenario. I understood that this is
>> probably the
>> > > >> main use case we are trying to cover with this AIP, thus I used it
>> as
>> > > >> example and mentioned it multiple times across the AIP.
>> > > >>
>> > > >> Thanks again for your time :)
>> > > >>
>> > > >>
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
>> > > >>
>> > > >> Vincent
>> > > >>
>> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
>> > > >> > Thanks a lot all for the comments, this is very much
>> appreciated! I
>> > > >> received many comments from this thread and in confluence, thanks
>> again.
>> > > >> I'll try to address them all in the AIP and will send an email in
>> this
>> > > >> thread once done. I will most likely revisit the push-based
>> approach
>> > > given
>> > > >> the number of concerns I received, thanks Jarek for proposing
>> another
>> > > >> solution, I'll probably go down that path.
>> > > >> >
>> > > >> > One follow-up question Vikram.
>> > > >> >
>> > > >> > > The bespoke triggerer approach completely makes sense for the
>> long
>> > > >> tail here, but can we do better for the 20% of scenarios which
>> cover
>> > > well
>> > > >> over 80% of usage here is the question in my mind. Or, are you
>> thinking
>> > > of
>> > > >> those as being covered in the "push" model?
>> > > >> >
>> > > >> > Could you share more details about what is this "20% of scenarios
>> > > which
>> > > >> cover well over 80% of usage" please?
>> > > >> >
>> > > >> > Vincent
>> > > >> >
>> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
>> > > >> > > Thanks Vincent for driving these, I have added my comments to
>> the
>> > > AIP
>> > > >> too.
>> > > >> > >
>> > > >> > > Regards,
>> > > >> > > Kaxil
>> > > >> > >
>> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
>> > > >> > > <Je...@de.bosch.com.invalid> wrote:
>> > > >> > >
>> > > >> > > > +1 on the comments of Vikram and Jarek, added main points on
>> > > >> confluence
>> > > >> > > >
>> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
>> > > >> > > > ________________________________
>> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
>> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
>> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
>> > > >> > > > Subject: Re: [DISCUSS] External event driven scheduling in
>> Airflow
>> > > >> > > >
>> > > >> > > > Vincent,
>> > > >> > > >
>> > > >> > > > Thanks for writing this up. The overview looks really good!
>> > > >> > > >
>> > > >> > > > I will leave my comments in the AIP as well, but at a high
>> level
>> > > >> they are
>> > > >> > > > both relatively focused on the "how", rather than the "what".
>> > > >> > > > With respect to the pull / polling approach, I completely
>> agree
>> > > >> that some
>> > > >> > > > incarnation of this is needed.
>> > > >> > > > I am less certain as to how on this part. The bespoke
>> triggerer
>> > > >> approach
>> > > >> > > > completely makes sense for the long tail here, but can we do
>> > > better
>> > > >> for the
>> > > >> > > > 20% of scenarios which cover well over 80% of usage here is
>> the
>> > > >> question in
>> > > >> > > > my mind. Or, are you thinking of those as being covered in
>> the
>> > > >> "push"
>> > > >> > > > model?
>> > > >> > > >
>> > > >> > > > Which leads to the "push" model approach.
>> > > >> > > > I am struggling with the same question that Jarek raised here
>> > > about
>> > > >> whether
>> > > >> > > > we need a new Airflow entity over and beyond the existing
>> REST API
>> > > >> for the
>> > > >> > > > same.
>> > > >> > > > I am concerned about this becoming a vector of attack on
>> Airflow.
>> > > >> > > > I see that this is a hot topic of discussion in the
>> Confluence doc
>> > > >> as well,
>> > > >> > > > but wanted to summarize here as well, so it didn't get lost
>> in the
>> > > >> threads
>> > > >> > > > of comments.
>> > > >> > > >
>> > > >> > > > Best regards,
>> > > >> > > > Vikram
>> > > >> > > >
>> > > >> > > >
>> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
>> jarek@potiuk.com>
>> > > >> wrote:
>> > > >> > > >
>> > > >> > > > > Thanks Vincent. I took a look and I have a general
>> comment. I
>> > > >> > > > > strongly think external driven scheduling is really needed
>> -
>> > > >> especially,
>> > > >> > > > it
>> > > >> > > > > should be much easier for a user to "plug-in" such an
>> external
>> > > >> event to
>> > > >> > > > > Airflow. And there are two parts of it - as correctly
>> stated
>> > > >> there - pull
>> > > >> > > > > and push.
>> > > >> > > > >
>> > > >> > > > > For the pull - I think it would be great to have a kind of
>> > > >> specialized
>> > > >> > > > > Triggers that will be started when DAG is parsed - and
>> those
>> > > >> Triggers
>> > > >> > > > could
>> > > >> > > > > generate the events for DAGs. I think basically that's all
>> that
>> > > is
>> > > >> > > > needed,
>> > > >> > > > > for example I imagine a pubsub trigger that will subscribe
>> to
>> > > >> messages
>> > > >> > > > > coming on the pubsub queue and fire "Asset" event when a
>> message
>> > > >> is
>> > > >> > > > > received. Not much controversy there - I am not sure about
>> the
>> > > >> polling
>> > > >> > > > > thing , because I've always believed that when
>> "asyncio-native"
>> > > >> Trigger
>> > > >> > > > is
>> > > >> > > > > run in the asyncio event loop, we do not "poll" every
>> second or
>> > > >> so (but
>> > > >> > > > > maybe this is just coming from some specific triggers that
>> > > >> actually do
>> > > >> > > > > such regular poll. But yes - there are polls like running
>> > > select
>> > > >> on the
>> > > >> > > > DB
>> > > >> > > > > that cannot be easily "async-ed" so having a configurable
>> > > polling
>> > > >> time
>> > > >> > > > > would be good there (but I am not sure maybe it's even
>> possible
>> > > >> today). I
>> > > >> > > > > think this would be really great if we have that option,
>> because
>> > > >> it makes
>> > > >> > > > > it much easier to set up the authorization for Airlfow
>> users -
>> > > >> rather
>> > > >> > > > than
>> > > >> > > > > setting up authorization and REST calls coming from an
>> external
>> > > >> system,
>> > > >> > > > we
>> > > >> > > > > can utilize Connections of Airlfow to authorize such a
>> Trigger
>> > > to
>> > > >> > > > subscribe
>> > > >> > > > > to events.
>> > > >> > > > >
>> > > >> > > > > For the push proposal - as I read the proposal, the main
>> point
>> > > >> behind it
>> > > >> > > > > is rather than users having to write "Airflow" way of
>> triggering
>> > > >> events
>> > > >> > > > and
>> > > >> > > > > configuring authentication (using REST API) to generate
>> asset
>> > > >> events, is
>> > > >> > > > to
>> > > >> > > > > make Airflow natively understand external ways of pushing
>> - and
>> > > >> > > > effectively
>> > > >> > > > > authorizing and mapping such incoming unauthorized
>> requests into
>> > > >> event
>> > > >> > > > that
>> > > >> > > > > could be generated by an API REST call.
>> > > >> > > > > I am not really sure honestly if this is something that we
>> want
>> > > as
>> > > >> > > > > "running" in airlfow as an endpoint. I'd say such an
>> > > unauthorised
>> > > >> > > > endpoint
>> > > >> > > > > is probably not a good idea - for a variety of reasons,
>> mostly
>> > > >> security.
>> > > >> > > > > And as I understand the goal is that users can easily
>> point at
>> > > >> > > > "3rd-party"
>> > > >> > > > > notification to Airflow and get the event generated.
>> > > >> > > > >
>> > > >> > > > > My feeling is that while this is needed - it should be
>> > > >> externalised from
>> > > >> > > > > airlfow webserver. The authorization has to be set up
>> anyway
>> > > >> > > > additionally -
>> > > >> > > > > unlike in "poll" case - we cannot use Connections for
>> > > authorizing
>> > > >> > > > (because
>> > > >> > > > > it's not Airlfow that authorizes in an external system -
>> it's
>> > > the
>> > > >> other
>> > > >> > > > way
>> > > >> > > > > round). So we have to anyhow setup "something extra" in
>> Airflow
>> > > to
>> > > >> > > > > authorize the external system. Which could be what we have
>> now -
>> > > >> user
>> > > >> > > > that
>> > > >> > > > > allows us to trigger the event. Which means that our REST
>> API
>> > > >> could
>> > > >> > > > > potentially be used the same way it is now, but we will
>> need
>> > > >> "something"
>> > > >> > > > > (library, lambda function etc.) that users could easily
>> setup in
>> > > >> the
>> > > >> > > > > external system to map whatever trigger they generate
>> natively
>> > > >> (say S3
>> > > >> > > > file
>> > > >> > > > > created) to Airflow REST API.
>> > > >> > > > >
>> > > >> > > > > As I see it - this is quite often used (and very
>> practical, that
>> > > >> you
>> > > >> > > > deploy
>> > > >> > > > > a cloud function or lambda that subscribes on the event
>> received
>> > > >> when
>> > > >> > > > > S3/GCS is created. So it would be on the user to deploy
>> such a
>> > > >> lambda -
>> > > >> > > > but
>> > > >> > > > > we **could** provide a library of those: say s3 lambda, gcp
>> > > cloud
>> > > >> > > > function
>> > > >> > > > > in respective providers - with documentation how to set
>> them up,
>> > > >> and how
>> > > >> > > > to
>> > > >> > > > > configure authorization and we would be generally "done".
>> I am
>> > > >> just not
>> > > >> > > > > sure if we need a new entity in Airflow for that (Event
>> > > >> receiver). It
>> > > >> > > > feels
>> > > >> > > > > like it asks Airflow to take more responsibility, when we
>> all
>> > > >> think on
>> > > >> > > > what
>> > > >> > > > > to "remove" from Airflow rather than "add" to it -
>> especially
>> > > >> when it
>> > > >> > > > comes
>> > > >> > > > > to external integrations. It feels to me that Airflow
>> should
>> > > make
>> > > >> it easy
>> > > >> > > > > to be triggered by such an external system and make it
>> easy to
>> > > >> "map" to
>> > > >> > > > the
>> > > >> > > > > way we expect to get events triggered, but this should be
>> done
>> > > >> outside of
>> > > >> > > > > Airflow. If the users can easily find in our docs when they
>> > > >> search "what
>> > > >> > > > do
>> > > >> > > > > I do to externally trigger Airflow on S3 change": either a)
>> > > >> configure
>> > > >> > > > > polling in airflow using s3 Connection, or b) "create a
>> user +
>> > > >> deploy
>> > > >> > > > this
>> > > >> > > > > lambda with those parameters" - that is "easy enough" and
>> very
>> > > >> practical
>> > > >> > > > > as well.
>> > > >> > > > >
>> > > >> > > > > But maybe I am not seeing the whole picture and the real
>> problem
>> > > >> it's
>> > > >> > > > > solving - so take it as a "first review pass" and "guts
>> > > feeling".
>> > > >> > > > >
>> > > >> > > > > J.
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > >
>> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
>> > > >> > > > <vincbeck@amazon.com.invalid
>> > > >> > > > > >
>> > > >> > > > > wrote:
>> > > >> > > > >
>> > > >> > > > > > Hello everyone,
>> > > >> > > > > >
>> > > >> > > > > > I created a draft AIP regarding "External event driven
>> > > >> scheduling in
>> > > >> > > > > > Airflow". This proposal is about adding capability in
>> Airflow
>> > > to
>> > > >> > > > schedule
>> > > >> > > > > > DAGs based on external events. Here are some examples of
>> such
>> > > >> external
>> > > >> > > > > > events:
>> > > >> > > > > > - A user signs up to one of the user pool defined in my
>> cloud
>> > > >> provider
>> > > >> > > > > > - One of the databases used in my company has been
>> updated
>> > > >> > > > > > - A job in my cloud provider has been executed
>> successfully
>> > > >> > > > > >
>> > > >> > > > > > The intent of this AIP is to leverage datasets (which
>> will be
>> > > >> soon
>> > > >> > > > > assets)
>> > > >> > > > > > and update them based on external events. I would like to
>> > > >> propose this
>> > > >> > > > > AIP
>> > > >> > > > > > for discussion and more importantly, hear some feedbacks
>> from
>> > > >> you :)
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > >
>> > > >>
>> > >
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
>> > > >> > > > <
>> > > >> > > >
>> > > >>
>> > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
>> > > >> > > > >
>> > > >> > > > > >
>> > > >> > > > > > Vincent
>> > > >> > > > > >
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >> >
>> ---------------------------------------------------------------------
>> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
>> > > >> >
>> > > >> >
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
>> > > >>
>> > > >>
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
I agree with both of you that it is indeed a good idea and that it can be
added in Future work -- doesn't need to be part of this AIP.
Thanks for the interest. I was not aware of such feature and this looks
> really cool! I definitely think that can be useful for Airflow, especially
> for testing when you can easily replay events received in the past.
> However, I do not think it should be part of the AIP and, as you mentioned,
> if should be a future work or a follow-up item of the AIP. Please let me
> know if you (or anyone) disagree with this and we can talk about it.
> Otherwise I'll update the future work section of the AIP and mention this
> archive and replay feature.
On Thu, 1 Aug 2024 at 16:11, Vincent Beck <vi...@apache.org> wrote:
> Hey Pavan,
>
> Thanks for the interest. I was not aware of such feature and this looks
> really cool! I definitely think that can be useful for Airflow, especially
> for testing when you can easily replay events received in the past.
> However, I do not think it should be part of the AIP and, as you mentioned,
> if should be a future work or a follow-up item of the AIP. Please let me
> know if you (or anyone) disagree with this and we can talk about it.
> Otherwise I'll update the future work section of the AIP and mention this
> archive and replay feature.
>
> On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> > Thanks Vincent, I took a look , this is really good. Don't have access to
> > the confluence page to comment :) so adding it here.
> >
> > As events arrive-->do somework-->end.
> >
> > So I'm uncertain if my comment pertains to the current poll/push model or
> > if it fits part of future work(seen event batching ).
> >
> > Have you given any thought to the event archival mechanism and event
> > replay? This could significantly aid in testing and recovery of workflow
> > and testing new functionality with events by just replay the events. The
> > archival mechanism I am referring to is similar to today in AWS we have
> > Event Bridge Archive and Replay.
> >
> > Regards,
> > Pavan
> >
> > On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > I actually did manage to take a look, thanks for the work. I am +1 on
> the
> > > poll-based approach -- left a comment on the push-based: I am not sure
> of
> > > why we need a function since create asset event API endpoint should
> have
> > > all info needed for what the Asset was.
> > >
> > > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com> wrote:
> > >
> > > > Thanks Vincent, I will take a look again tomorrow.
> > > >
> > > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org>
> wrote:
> > > >
> > > >> Hi everyone,
> > > >>
> > > >> I updated the AIP-82 given the different comments and concerns I
> > > >> received. I also tried to reply to all comments individually. I
> would
> > > >> really appreciate if you can do a second pass and let me know what
> you
> > > >> think. Overall, this is what I changed in the AIP:
> > > >>
> > > >> - Push based event-driven scheduling. I updated this section
> entirely
> > > >> because I received many concerns about the previous proposal. The
> > > overall
> > > >> idea now is to leverage the create asset event API endpoint to send
> > > >> notifications from external (e.g. cloud provider) to Airflow
> > > environment.
> > > >>
> > > >> - I updated the poll based event-driven scheduling DAG author
> experience
> > > >> to use a message queue scenario. I understood that this is probably
> the
> > > >> main use case we are trying to cover with this AIP, thus I used it
> as
> > > >> example and mentioned it multiple times across the AIP.
> > > >>
> > > >> Thanks again for your time :)
> > > >>
> > > >>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > >>
> > > >> Vincent
> > > >>
> > > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > > >> > Thanks a lot all for the comments, this is very much appreciated!
> I
> > > >> received many comments from this thread and in confluence, thanks
> again.
> > > >> I'll try to address them all in the AIP and will send an email in
> this
> > > >> thread once done. I will most likely revisit the push-based approach
> > > given
> > > >> the number of concerns I received, thanks Jarek for proposing
> another
> > > >> solution, I'll probably go down that path.
> > > >> >
> > > >> > One follow-up question Vikram.
> > > >> >
> > > >> > > The bespoke triggerer approach completely makes sense for the
> long
> > > >> tail here, but can we do better for the 20% of scenarios which cover
> > > well
> > > >> over 80% of usage here is the question in my mind. Or, are you
> thinking
> > > of
> > > >> those as being covered in the "push" model?
> > > >> >
> > > >> > Could you share more details about what is this "20% of scenarios
> > > which
> > > >> cover well over 80% of usage" please?
> > > >> >
> > > >> > Vincent
> > > >> >
> > > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > >> > > Thanks Vincent for driving these, I have added my comments to
> the
> > > AIP
> > > >> too.
> > > >> > >
> > > >> > > Regards,
> > > >> > > Kaxil
> > > >> > >
> > > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
> > > >> > > <Je...@de.bosch.com.invalid> wrote:
> > > >> > >
> > > >> > > > +1 on the comments of Vikram and Jarek, added main points on
> > > >> confluence
> > > >> > > >
> > > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > >> > > > ________________________________
> > > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > >> > > > Subject: Re: [DISCUSS] External event driven scheduling in
> Airflow
> > > >> > > >
> > > >> > > > Vincent,
> > > >> > > >
> > > >> > > > Thanks for writing this up. The overview looks really good!
> > > >> > > >
> > > >> > > > I will leave my comments in the AIP as well, but at a high
> level
> > > >> they are
> > > >> > > > both relatively focused on the "how", rather than the "what".
> > > >> > > > With respect to the pull / polling approach, I completely
> agree
> > > >> that some
> > > >> > > > incarnation of this is needed.
> > > >> > > > I am less certain as to how on this part. The bespoke
> triggerer
> > > >> approach
> > > >> > > > completely makes sense for the long tail here, but can we do
> > > better
> > > >> for the
> > > >> > > > 20% of scenarios which cover well over 80% of usage here is
> the
> > > >> question in
> > > >> > > > my mind. Or, are you thinking of those as being covered in the
> > > >> "push"
> > > >> > > > model?
> > > >> > > >
> > > >> > > > Which leads to the "push" model approach.
> > > >> > > > I am struggling with the same question that Jarek raised here
> > > about
> > > >> whether
> > > >> > > > we need a new Airflow entity over and beyond the existing
> REST API
> > > >> for the
> > > >> > > > same.
> > > >> > > > I am concerned about this becoming a vector of attack on
> Airflow.
> > > >> > > > I see that this is a hot topic of discussion in the
> Confluence doc
> > > >> as well,
> > > >> > > > but wanted to summarize here as well, so it didn't get lost
> in the
> > > >> threads
> > > >> > > > of comments.
> > > >> > > >
> > > >> > > > Best regards,
> > > >> > > > Vikram
> > > >> > > >
> > > >> > > >
> > > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <
> jarek@potiuk.com>
> > > >> wrote:
> > > >> > > >
> > > >> > > > > Thanks Vincent. I took a look and I have a general comment.
> I
> > > >> > > > > strongly think external driven scheduling is really needed -
> > > >> especially,
> > > >> > > > it
> > > >> > > > > should be much easier for a user to "plug-in" such an
> external
> > > >> event to
> > > >> > > > > Airflow. And there are two parts of it - as correctly stated
> > > >> there - pull
> > > >> > > > > and push.
> > > >> > > > >
> > > >> > > > > For the pull - I think it would be great to have a kind of
> > > >> specialized
> > > >> > > > > Triggers that will be started when DAG is parsed - and those
> > > >> Triggers
> > > >> > > > could
> > > >> > > > > generate the events for DAGs. I think basically that's all
> that
> > > is
> > > >> > > > needed,
> > > >> > > > > for example I imagine a pubsub trigger that will subscribe
> to
> > > >> messages
> > > >> > > > > coming on the pubsub queue and fire "Asset" event when a
> message
> > > >> is
> > > >> > > > > received. Not much controversy there - I am not sure about
> the
> > > >> polling
> > > >> > > > > thing , because I've always believed that when
> "asyncio-native"
> > > >> Trigger
> > > >> > > > is
> > > >> > > > > run in the asyncio event loop, we do not "poll" every
> second or
> > > >> so (but
> > > >> > > > > maybe this is just coming from some specific triggers that
> > > >> actually do
> > > >> > > > > such regular poll. But yes - there are polls like running
> > > select
> > > >> on the
> > > >> > > > DB
> > > >> > > > > that cannot be easily "async-ed" so having a configurable
> > > polling
> > > >> time
> > > >> > > > > would be good there (but I am not sure maybe it's even
> possible
> > > >> today). I
> > > >> > > > > think this would be really great if we have that option,
> because
> > > >> it makes
> > > >> > > > > it much easier to set up the authorization for Airlfow
> users -
> > > >> rather
> > > >> > > > than
> > > >> > > > > setting up authorization and REST calls coming from an
> external
> > > >> system,
> > > >> > > > we
> > > >> > > > > can utilize Connections of Airlfow to authorize such a
> Trigger
> > > to
> > > >> > > > subscribe
> > > >> > > > > to events.
> > > >> > > > >
> > > >> > > > > For the push proposal - as I read the proposal, the main
> point
> > > >> behind it
> > > >> > > > > is rather than users having to write "Airflow" way of
> triggering
> > > >> events
> > > >> > > > and
> > > >> > > > > configuring authentication (using REST API) to generate
> asset
> > > >> events, is
> > > >> > > > to
> > > >> > > > > make Airflow natively understand external ways of pushing -
> and
> > > >> > > > effectively
> > > >> > > > > authorizing and mapping such incoming unauthorized requests
> into
> > > >> event
> > > >> > > > that
> > > >> > > > > could be generated by an API REST call.
> > > >> > > > > I am not really sure honestly if this is something that we
> want
> > > as
> > > >> > > > > "running" in airlfow as an endpoint. I'd say such an
> > > unauthorised
> > > >> > > > endpoint
> > > >> > > > > is probably not a good idea - for a variety of reasons,
> mostly
> > > >> security.
> > > >> > > > > And as I understand the goal is that users can easily point
> at
> > > >> > > > "3rd-party"
> > > >> > > > > notification to Airflow and get the event generated.
> > > >> > > > >
> > > >> > > > > My feeling is that while this is needed - it should be
> > > >> externalised from
> > > >> > > > > airlfow webserver. The authorization has to be set up anyway
> > > >> > > > additionally -
> > > >> > > > > unlike in "poll" case - we cannot use Connections for
> > > authorizing
> > > >> > > > (because
> > > >> > > > > it's not Airlfow that authorizes in an external system -
> it's
> > > the
> > > >> other
> > > >> > > > way
> > > >> > > > > round). So we have to anyhow setup "something extra" in
> Airflow
> > > to
> > > >> > > > > authorize the external system. Which could be what we have
> now -
> > > >> user
> > > >> > > > that
> > > >> > > > > allows us to trigger the event. Which means that our REST
> API
> > > >> could
> > > >> > > > > potentially be used the same way it is now, but we will need
> > > >> "something"
> > > >> > > > > (library, lambda function etc.) that users could easily
> setup in
> > > >> the
> > > >> > > > > external system to map whatever trigger they generate
> natively
> > > >> (say S3
> > > >> > > > file
> > > >> > > > > created) to Airflow REST API.
> > > >> > > > >
> > > >> > > > > As I see it - this is quite often used (and very practical,
> that
> > > >> you
> > > >> > > > deploy
> > > >> > > > > a cloud function or lambda that subscribes on the event
> received
> > > >> when
> > > >> > > > > S3/GCS is created. So it would be on the user to deploy
> such a
> > > >> lambda -
> > > >> > > > but
> > > >> > > > > we **could** provide a library of those: say s3 lambda, gcp
> > > cloud
> > > >> > > > function
> > > >> > > > > in respective providers - with documentation how to set
> them up,
> > > >> and how
> > > >> > > > to
> > > >> > > > > configure authorization and we would be generally "done". I
> am
> > > >> just not
> > > >> > > > > sure if we need a new entity in Airflow for that (Event
> > > >> receiver). It
> > > >> > > > feels
> > > >> > > > > like it asks Airflow to take more responsibility, when we
> all
> > > >> think on
> > > >> > > > what
> > > >> > > > > to "remove" from Airflow rather than "add" to it -
> especially
> > > >> when it
> > > >> > > > comes
> > > >> > > > > to external integrations. It feels to me that Airflow should
> > > make
> > > >> it easy
> > > >> > > > > to be triggered by such an external system and make it easy
> to
> > > >> "map" to
> > > >> > > > the
> > > >> > > > > way we expect to get events triggered, but this should be
> done
> > > >> outside of
> > > >> > > > > Airflow. If the users can easily find in our docs when they
> > > >> search "what
> > > >> > > > do
> > > >> > > > > I do to externally trigger Airflow on S3 change": either a)
> > > >> configure
> > > >> > > > > polling in airflow using s3 Connection, or b) "create a
> user +
> > > >> deploy
> > > >> > > > this
> > > >> > > > > lambda with those parameters" - that is "easy enough" and
> very
> > > >> practical
> > > >> > > > > as well.
> > > >> > > > >
> > > >> > > > > But maybe I am not seeing the whole picture and the real
> problem
> > > >> it's
> > > >> > > > > solving - so take it as a "first review pass" and "guts
> > > feeling".
> > > >> > > > >
> > > >> > > > > J.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > >> > > > <vincbeck@amazon.com.invalid
> > > >> > > > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hello everyone,
> > > >> > > > > >
> > > >> > > > > > I created a draft AIP regarding "External event driven
> > > >> scheduling in
> > > >> > > > > > Airflow". This proposal is about adding capability in
> Airflow
> > > to
> > > >> > > > schedule
> > > >> > > > > > DAGs based on external events. Here are some examples of
> such
> > > >> external
> > > >> > > > > > events:
> > > >> > > > > > - A user signs up to one of the user pool defined in my
> cloud
> > > >> provider
> > > >> > > > > > - One of the databases used in my company has been updated
> > > >> > > > > > - A job in my cloud provider has been executed
> successfully
> > > >> > > > > >
> > > >> > > > > > The intent of this AIP is to leverage datasets (which
> will be
> > > >> soon
> > > >> > > > > assets)
> > > >> > > > > > and update them based on external events. I would like to
> > > >> propose this
> > > >> > > > > AIP
> > > >> > > > > > for discussion and more importantly, hear some feedbacks
> from
> > > >> you :)
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >>
> > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > >> > > > <
> > > >> > > >
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > Vincent
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> ---------------------------------------------------------------------
> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > > >>
> > > >>
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vincent Beck <vi...@apache.org>.
Hey Pavan,
Thanks for the interest. I was not aware of such feature and this looks really cool! I definitely think that can be useful for Airflow, especially for testing when you can easily replay events received in the past. However, I do not think it should be part of the AIP and, as you mentioned, if should be a future work or a follow-up item of the AIP. Please let me know if you (or anyone) disagree with this and we can talk about it. Otherwise I'll update the future work section of the AIP and mention this archive and replay feature.
On 2024/08/01 01:21:58 Pavankumar Gopidesu wrote:
> Thanks Vincent, I took a look , this is really good. Don't have access to
> the confluence page to comment :) so adding it here.
>
> As events arrive-->do somework-->end.
>
> So I'm uncertain if my comment pertains to the current poll/push model or
> if it fits part of future work(seen event batching ).
>
> Have you given any thought to the event archival mechanism and event
> replay? This could significantly aid in testing and recovery of workflow
> and testing new functionality with events by just replay the events. The
> archival mechanism I am referring to is similar to today in AWS we have
> Event Bridge Archive and Replay.
>
> Regards,
> Pavan
>
> On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > I actually did manage to take a look, thanks for the work. I am +1 on the
> > poll-based approach -- left a comment on the push-based: I am not sure of
> > why we need a function since create asset event API endpoint should have
> > all info needed for what the Asset was.
> >
> > On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > Thanks Vincent, I will take a look again tomorrow.
> > >
> > > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org> wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I updated the AIP-82 given the different comments and concerns I
> > >> received. I also tried to reply to all comments individually. I would
> > >> really appreciate if you can do a second pass and let me know what you
> > >> think. Overall, this is what I changed in the AIP:
> > >>
> > >> - Push based event-driven scheduling. I updated this section entirely
> > >> because I received many concerns about the previous proposal. The
> > overall
> > >> idea now is to leverage the create asset event API endpoint to send
> > >> notifications from external (e.g. cloud provider) to Airflow
> > environment.
> > >>
> > >> - I updated the poll based event-driven scheduling DAG author experience
> > >> to use a message queue scenario. I understood that this is probably the
> > >> main use case we are trying to cover with this AIP, thus I used it as
> > >> example and mentioned it multiple times across the AIP.
> > >>
> > >> Thanks again for your time :)
> > >>
> > >>
> > >>
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > >>
> > >> Vincent
> > >>
> > >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > >> > Thanks a lot all for the comments, this is very much appreciated! I
> > >> received many comments from this thread and in confluence, thanks again.
> > >> I'll try to address them all in the AIP and will send an email in this
> > >> thread once done. I will most likely revisit the push-based approach
> > given
> > >> the number of concerns I received, thanks Jarek for proposing another
> > >> solution, I'll probably go down that path.
> > >> >
> > >> > One follow-up question Vikram.
> > >> >
> > >> > > The bespoke triggerer approach completely makes sense for the long
> > >> tail here, but can we do better for the 20% of scenarios which cover
> > well
> > >> over 80% of usage here is the question in my mind. Or, are you thinking
> > of
> > >> those as being covered in the "push" model?
> > >> >
> > >> > Could you share more details about what is this "20% of scenarios
> > which
> > >> cover well over 80% of usage" please?
> > >> >
> > >> > Vincent
> > >> >
> > >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > >> > > Thanks Vincent for driving these, I have added my comments to the
> > AIP
> > >> too.
> > >> > >
> > >> > > Regards,
> > >> > > Kaxil
> > >> > >
> > >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
> > >> > > <Je...@de.bosch.com.invalid> wrote:
> > >> > >
> > >> > > > +1 on the comments of Vikram and Jarek, added main points on
> > >> confluence
> > >> > > >
> > >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > >> > > > ________________________________
> > >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > >> > > > Subject: Re: [DISCUSS] External event driven scheduling in Airflow
> > >> > > >
> > >> > > > Vincent,
> > >> > > >
> > >> > > > Thanks for writing this up. The overview looks really good!
> > >> > > >
> > >> > > > I will leave my comments in the AIP as well, but at a high level
> > >> they are
> > >> > > > both relatively focused on the "how", rather than the "what".
> > >> > > > With respect to the pull / polling approach, I completely agree
> > >> that some
> > >> > > > incarnation of this is needed.
> > >> > > > I am less certain as to how on this part. The bespoke triggerer
> > >> approach
> > >> > > > completely makes sense for the long tail here, but can we do
> > better
> > >> for the
> > >> > > > 20% of scenarios which cover well over 80% of usage here is the
> > >> question in
> > >> > > > my mind. Or, are you thinking of those as being covered in the
> > >> "push"
> > >> > > > model?
> > >> > > >
> > >> > > > Which leads to the "push" model approach.
> > >> > > > I am struggling with the same question that Jarek raised here
> > about
> > >> whether
> > >> > > > we need a new Airflow entity over and beyond the existing REST API
> > >> for the
> > >> > > > same.
> > >> > > > I am concerned about this becoming a vector of attack on Airflow.
> > >> > > > I see that this is a hot topic of discussion in the Confluence doc
> > >> as well,
> > >> > > > but wanted to summarize here as well, so it didn't get lost in the
> > >> threads
> > >> > > > of comments.
> > >> > > >
> > >> > > > Best regards,
> > >> > > > Vikram
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com>
> > >> wrote:
> > >> > > >
> > >> > > > > Thanks Vincent. I took a look and I have a general comment. I
> > >> > > > > strongly think external driven scheduling is really needed -
> > >> especially,
> > >> > > > it
> > >> > > > > should be much easier for a user to "plug-in" such an external
> > >> event to
> > >> > > > > Airflow. And there are two parts of it - as correctly stated
> > >> there - pull
> > >> > > > > and push.
> > >> > > > >
> > >> > > > > For the pull - I think it would be great to have a kind of
> > >> specialized
> > >> > > > > Triggers that will be started when DAG is parsed - and those
> > >> Triggers
> > >> > > > could
> > >> > > > > generate the events for DAGs. I think basically that's all that
> > is
> > >> > > > needed,
> > >> > > > > for example I imagine a pubsub trigger that will subscribe to
> > >> messages
> > >> > > > > coming on the pubsub queue and fire "Asset" event when a message
> > >> is
> > >> > > > > received. Not much controversy there - I am not sure about the
> > >> polling
> > >> > > > > thing , because I've always believed that when "asyncio-native"
> > >> Trigger
> > >> > > > is
> > >> > > > > run in the asyncio event loop, we do not "poll" every second or
> > >> so (but
> > >> > > > > maybe this is just coming from some specific triggers that
> > >> actually do
> > >> > > > > such regular poll. But yes - there are polls like running
> > select
> > >> on the
> > >> > > > DB
> > >> > > > > that cannot be easily "async-ed" so having a configurable
> > polling
> > >> time
> > >> > > > > would be good there (but I am not sure maybe it's even possible
> > >> today). I
> > >> > > > > think this would be really great if we have that option, because
> > >> it makes
> > >> > > > > it much easier to set up the authorization for Airlfow users -
> > >> rather
> > >> > > > than
> > >> > > > > setting up authorization and REST calls coming from an external
> > >> system,
> > >> > > > we
> > >> > > > > can utilize Connections of Airlfow to authorize such a Trigger
> > to
> > >> > > > subscribe
> > >> > > > > to events.
> > >> > > > >
> > >> > > > > For the push proposal - as I read the proposal, the main point
> > >> behind it
> > >> > > > > is rather than users having to write "Airflow" way of triggering
> > >> events
> > >> > > > and
> > >> > > > > configuring authentication (using REST API) to generate asset
> > >> events, is
> > >> > > > to
> > >> > > > > make Airflow natively understand external ways of pushing - and
> > >> > > > effectively
> > >> > > > > authorizing and mapping such incoming unauthorized requests into
> > >> event
> > >> > > > that
> > >> > > > > could be generated by an API REST call.
> > >> > > > > I am not really sure honestly if this is something that we want
> > as
> > >> > > > > "running" in airlfow as an endpoint. I'd say such an
> > unauthorised
> > >> > > > endpoint
> > >> > > > > is probably not a good idea - for a variety of reasons, mostly
> > >> security.
> > >> > > > > And as I understand the goal is that users can easily point at
> > >> > > > "3rd-party"
> > >> > > > > notification to Airflow and get the event generated.
> > >> > > > >
> > >> > > > > My feeling is that while this is needed - it should be
> > >> externalised from
> > >> > > > > airlfow webserver. The authorization has to be set up anyway
> > >> > > > additionally -
> > >> > > > > unlike in "poll" case - we cannot use Connections for
> > authorizing
> > >> > > > (because
> > >> > > > > it's not Airlfow that authorizes in an external system - it's
> > the
> > >> other
> > >> > > > way
> > >> > > > > round). So we have to anyhow setup "something extra" in Airflow
> > to
> > >> > > > > authorize the external system. Which could be what we have now -
> > >> user
> > >> > > > that
> > >> > > > > allows us to trigger the event. Which means that our REST API
> > >> could
> > >> > > > > potentially be used the same way it is now, but we will need
> > >> "something"
> > >> > > > > (library, lambda function etc.) that users could easily setup in
> > >> the
> > >> > > > > external system to map whatever trigger they generate natively
> > >> (say S3
> > >> > > > file
> > >> > > > > created) to Airflow REST API.
> > >> > > > >
> > >> > > > > As I see it - this is quite often used (and very practical, that
> > >> you
> > >> > > > deploy
> > >> > > > > a cloud function or lambda that subscribes on the event received
> > >> when
> > >> > > > > S3/GCS is created. So it would be on the user to deploy such a
> > >> lambda -
> > >> > > > but
> > >> > > > > we **could** provide a library of those: say s3 lambda, gcp
> > cloud
> > >> > > > function
> > >> > > > > in respective providers - with documentation how to set them up,
> > >> and how
> > >> > > > to
> > >> > > > > configure authorization and we would be generally "done". I am
> > >> just not
> > >> > > > > sure if we need a new entity in Airflow for that (Event
> > >> receiver). It
> > >> > > > feels
> > >> > > > > like it asks Airflow to take more responsibility, when we all
> > >> think on
> > >> > > > what
> > >> > > > > to "remove" from Airflow rather than "add" to it - especially
> > >> when it
> > >> > > > comes
> > >> > > > > to external integrations. It feels to me that Airflow should
> > make
> > >> it easy
> > >> > > > > to be triggered by such an external system and make it easy to
> > >> "map" to
> > >> > > > the
> > >> > > > > way we expect to get events triggered, but this should be done
> > >> outside of
> > >> > > > > Airflow. If the users can easily find in our docs when they
> > >> search "what
> > >> > > > do
> > >> > > > > I do to externally trigger Airflow on S3 change": either a)
> > >> configure
> > >> > > > > polling in airflow using s3 Connection, or b) "create a user +
> > >> deploy
> > >> > > > this
> > >> > > > > lambda with those parameters" - that is "easy enough" and very
> > >> practical
> > >> > > > > as well.
> > >> > > > >
> > >> > > > > But maybe I am not seeing the whole picture and the real problem
> > >> it's
> > >> > > > > solving - so take it as a "first review pass" and "guts
> > feeling".
> > >> > > > >
> > >> > > > > J.
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > >> > > > <vincbeck@amazon.com.invalid
> > >> > > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hello everyone,
> > >> > > > > >
> > >> > > > > > I created a draft AIP regarding "External event driven
> > >> scheduling in
> > >> > > > > > Airflow". This proposal is about adding capability in Airflow
> > to
> > >> > > > schedule
> > >> > > > > > DAGs based on external events. Here are some examples of such
> > >> external
> > >> > > > > > events:
> > >> > > > > > - A user signs up to one of the user pool defined in my cloud
> > >> provider
> > >> > > > > > - One of the databases used in my company has been updated
> > >> > > > > > - A job in my cloud provider has been executed successfully
> > >> > > > > >
> > >> > > > > > The intent of this AIP is to leverage datasets (which will be
> > >> soon
> > >> > > > > assets)
> > >> > > > > > and update them based on external events. I would like to
> > >> propose this
> > >> > > > > AIP
> > >> > > > > > for discussion and more importantly, hear some feedbacks from
> > >> you :)
> > >> > > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >>
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > >> > > > <
> > >> > > >
> > >>
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > >> > > > >
> > >> > > > > >
> > >> > > > > > Vincent
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> > ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> >
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> For additional commands, e-mail: dev-help@airflow.apache.org
> > >>
> > >>
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Thanks Vincent, I took a look , this is really good. Don't have access to
the confluence page to comment :) so adding it here.
As events arrive-->do somework-->end.
So I'm uncertain if my comment pertains to the current poll/push model or
if it fits part of future work(seen event batching ).
Have you given any thought to the event archival mechanism and event
replay? This could significantly aid in testing and recovery of workflow
and testing new functionality with events by just replay the events. The
archival mechanism I am referring to is similar to today in AWS we have
Event Bridge Archive and Replay.
Regards,
Pavan
On Thu, Aug 1, 2024 at 1:29 AM Kaxil Naik <ka...@gmail.com> wrote:
> I actually did manage to take a look, thanks for the work. I am +1 on the
> poll-based approach -- left a comment on the push-based: I am not sure of
> why we need a function since create asset event API endpoint should have
> all info needed for what the Asset was.
>
> On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com> wrote:
>
> > Thanks Vincent, I will take a look again tomorrow.
> >
> > On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org> wrote:
> >
> >> Hi everyone,
> >>
> >> I updated the AIP-82 given the different comments and concerns I
> >> received. I also tried to reply to all comments individually. I would
> >> really appreciate if you can do a second pass and let me know what you
> >> think. Overall, this is what I changed in the AIP:
> >>
> >> - Push based event-driven scheduling. I updated this section entirely
> >> because I received many concerns about the previous proposal. The
> overall
> >> idea now is to leverage the create asset event API endpoint to send
> >> notifications from external (e.g. cloud provider) to Airflow
> environment.
> >>
> >> - I updated the poll based event-driven scheduling DAG author experience
> >> to use a message queue scenario. I understood that this is probably the
> >> main use case we are trying to cover with this AIP, thus I used it as
> >> example and mentioned it multiple times across the AIP.
> >>
> >> Thanks again for your time :)
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> >>
> >> Vincent
> >>
> >> On 2024/07/29 15:58:23 Vincent Beck wrote:
> >> > Thanks a lot all for the comments, this is very much appreciated! I
> >> received many comments from this thread and in confluence, thanks again.
> >> I'll try to address them all in the AIP and will send an email in this
> >> thread once done. I will most likely revisit the push-based approach
> given
> >> the number of concerns I received, thanks Jarek for proposing another
> >> solution, I'll probably go down that path.
> >> >
> >> > One follow-up question Vikram.
> >> >
> >> > > The bespoke triggerer approach completely makes sense for the long
> >> tail here, but can we do better for the 20% of scenarios which cover
> well
> >> over 80% of usage here is the question in my mind. Or, are you thinking
> of
> >> those as being covered in the "push" model?
> >> >
> >> > Could you share more details about what is this "20% of scenarios
> which
> >> cover well over 80% of usage" please?
> >> >
> >> > Vincent
> >> >
> >> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> >> > > Thanks Vincent for driving these, I have added my comments to the
> AIP
> >> too.
> >> > >
> >> > > Regards,
> >> > > Kaxil
> >> > >
> >> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
> >> > > <Je...@de.bosch.com.invalid> wrote:
> >> > >
> >> > > > +1 on the comments of Vikram and Jarek, added main points on
> >> confluence
> >> > > >
> >> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> >> > > > ________________________________
> >> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> >> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> >> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> >> > > > Subject: Re: [DISCUSS] External event driven scheduling in Airflow
> >> > > >
> >> > > > Vincent,
> >> > > >
> >> > > > Thanks for writing this up. The overview looks really good!
> >> > > >
> >> > > > I will leave my comments in the AIP as well, but at a high level
> >> they are
> >> > > > both relatively focused on the "how", rather than the "what".
> >> > > > With respect to the pull / polling approach, I completely agree
> >> that some
> >> > > > incarnation of this is needed.
> >> > > > I am less certain as to how on this part. The bespoke triggerer
> >> approach
> >> > > > completely makes sense for the long tail here, but can we do
> better
> >> for the
> >> > > > 20% of scenarios which cover well over 80% of usage here is the
> >> question in
> >> > > > my mind. Or, are you thinking of those as being covered in the
> >> "push"
> >> > > > model?
> >> > > >
> >> > > > Which leads to the "push" model approach.
> >> > > > I am struggling with the same question that Jarek raised here
> about
> >> whether
> >> > > > we need a new Airflow entity over and beyond the existing REST API
> >> for the
> >> > > > same.
> >> > > > I am concerned about this becoming a vector of attack on Airflow.
> >> > > > I see that this is a hot topic of discussion in the Confluence doc
> >> as well,
> >> > > > but wanted to summarize here as well, so it didn't get lost in the
> >> threads
> >> > > > of comments.
> >> > > >
> >> > > > Best regards,
> >> > > > Vikram
> >> > > >
> >> > > >
> >> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com>
> >> wrote:
> >> > > >
> >> > > > > Thanks Vincent. I took a look and I have a general comment. I
> >> > > > > strongly think external driven scheduling is really needed -
> >> especially,
> >> > > > it
> >> > > > > should be much easier for a user to "plug-in" such an external
> >> event to
> >> > > > > Airflow. And there are two parts of it - as correctly stated
> >> there - pull
> >> > > > > and push.
> >> > > > >
> >> > > > > For the pull - I think it would be great to have a kind of
> >> specialized
> >> > > > > Triggers that will be started when DAG is parsed - and those
> >> Triggers
> >> > > > could
> >> > > > > generate the events for DAGs. I think basically that's all that
> is
> >> > > > needed,
> >> > > > > for example I imagine a pubsub trigger that will subscribe to
> >> messages
> >> > > > > coming on the pubsub queue and fire "Asset" event when a message
> >> is
> >> > > > > received. Not much controversy there - I am not sure about the
> >> polling
> >> > > > > thing , because I've always believed that when "asyncio-native"
> >> Trigger
> >> > > > is
> >> > > > > run in the asyncio event loop, we do not "poll" every second or
> >> so (but
> >> > > > > maybe this is just coming from some specific triggers that
> >> actually do
> >> > > > > such regular poll. But yes - there are polls like running
> select
> >> on the
> >> > > > DB
> >> > > > > that cannot be easily "async-ed" so having a configurable
> polling
> >> time
> >> > > > > would be good there (but I am not sure maybe it's even possible
> >> today). I
> >> > > > > think this would be really great if we have that option, because
> >> it makes
> >> > > > > it much easier to set up the authorization for Airlfow users -
> >> rather
> >> > > > than
> >> > > > > setting up authorization and REST calls coming from an external
> >> system,
> >> > > > we
> >> > > > > can utilize Connections of Airlfow to authorize such a Trigger
> to
> >> > > > subscribe
> >> > > > > to events.
> >> > > > >
> >> > > > > For the push proposal - as I read the proposal, the main point
> >> behind it
> >> > > > > is rather than users having to write "Airflow" way of triggering
> >> events
> >> > > > and
> >> > > > > configuring authentication (using REST API) to generate asset
> >> events, is
> >> > > > to
> >> > > > > make Airflow natively understand external ways of pushing - and
> >> > > > effectively
> >> > > > > authorizing and mapping such incoming unauthorized requests into
> >> event
> >> > > > that
> >> > > > > could be generated by an API REST call.
> >> > > > > I am not really sure honestly if this is something that we want
> as
> >> > > > > "running" in airlfow as an endpoint. I'd say such an
> unauthorised
> >> > > > endpoint
> >> > > > > is probably not a good idea - for a variety of reasons, mostly
> >> security.
> >> > > > > And as I understand the goal is that users can easily point at
> >> > > > "3rd-party"
> >> > > > > notification to Airflow and get the event generated.
> >> > > > >
> >> > > > > My feeling is that while this is needed - it should be
> >> externalised from
> >> > > > > airlfow webserver. The authorization has to be set up anyway
> >> > > > additionally -
> >> > > > > unlike in "poll" case - we cannot use Connections for
> authorizing
> >> > > > (because
> >> > > > > it's not Airlfow that authorizes in an external system - it's
> the
> >> other
> >> > > > way
> >> > > > > round). So we have to anyhow setup "something extra" in Airflow
> to
> >> > > > > authorize the external system. Which could be what we have now -
> >> user
> >> > > > that
> >> > > > > allows us to trigger the event. Which means that our REST API
> >> could
> >> > > > > potentially be used the same way it is now, but we will need
> >> "something"
> >> > > > > (library, lambda function etc.) that users could easily setup in
> >> the
> >> > > > > external system to map whatever trigger they generate natively
> >> (say S3
> >> > > > file
> >> > > > > created) to Airflow REST API.
> >> > > > >
> >> > > > > As I see it - this is quite often used (and very practical, that
> >> you
> >> > > > deploy
> >> > > > > a cloud function or lambda that subscribes on the event received
> >> when
> >> > > > > S3/GCS is created. So it would be on the user to deploy such a
> >> lambda -
> >> > > > but
> >> > > > > we **could** provide a library of those: say s3 lambda, gcp
> cloud
> >> > > > function
> >> > > > > in respective providers - with documentation how to set them up,
> >> and how
> >> > > > to
> >> > > > > configure authorization and we would be generally "done". I am
> >> just not
> >> > > > > sure if we need a new entity in Airflow for that (Event
> >> receiver). It
> >> > > > feels
> >> > > > > like it asks Airflow to take more responsibility, when we all
> >> think on
> >> > > > what
> >> > > > > to "remove" from Airflow rather than "add" to it - especially
> >> when it
> >> > > > comes
> >> > > > > to external integrations. It feels to me that Airflow should
> make
> >> it easy
> >> > > > > to be triggered by such an external system and make it easy to
> >> "map" to
> >> > > > the
> >> > > > > way we expect to get events triggered, but this should be done
> >> outside of
> >> > > > > Airflow. If the users can easily find in our docs when they
> >> search "what
> >> > > > do
> >> > > > > I do to externally trigger Airflow on S3 change": either a)
> >> configure
> >> > > > > polling in airflow using s3 Connection, or b) "create a user +
> >> deploy
> >> > > > this
> >> > > > > lambda with those parameters" - that is "easy enough" and very
> >> practical
> >> > > > > as well.
> >> > > > >
> >> > > > > But maybe I am not seeing the whole picture and the real problem
> >> it's
> >> > > > > solving - so take it as a "first review pass" and "guts
> feeling".
> >> > > > >
> >> > > > > J.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> >> > > > <vincbeck@amazon.com.invalid
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hello everyone,
> >> > > > > >
> >> > > > > > I created a draft AIP regarding "External event driven
> >> scheduling in
> >> > > > > > Airflow". This proposal is about adding capability in Airflow
> to
> >> > > > schedule
> >> > > > > > DAGs based on external events. Here are some examples of such
> >> external
> >> > > > > > events:
> >> > > > > > - A user signs up to one of the user pool defined in my cloud
> >> provider
> >> > > > > > - One of the databases used in my company has been updated
> >> > > > > > - A job in my cloud provider has been executed successfully
> >> > > > > >
> >> > > > > > The intent of this AIP is to leverage datasets (which will be
> >> soon
> >> > > > > assets)
> >> > > > > > and update them based on external events. I would like to
> >> propose this
> >> > > > > AIP
> >> > > > > > for discussion and more importantly, hear some feedbacks from
> >> you :)
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> >> > > > <
> >> > > >
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> >> > > > >
> >> > > > > >
> >> > > > > > Vincent
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > For additional commands, e-mail: dev-help@airflow.apache.org
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> For additional commands, e-mail: dev-help@airflow.apache.org
> >>
> >>
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
I actually did manage to take a look, thanks for the work. I am +1 on the
poll-based approach -- left a comment on the push-based: I am not sure of
why we need a function since create asset event API endpoint should have
all info needed for what the Asset was.
On Thu, 1 Aug 2024 at 01:14, Kaxil Naik <ka...@gmail.com> wrote:
> Thanks Vincent, I will take a look again tomorrow.
>
> On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org> wrote:
>
>> Hi everyone,
>>
>> I updated the AIP-82 given the different comments and concerns I
>> received. I also tried to reply to all comments individually. I would
>> really appreciate if you can do a second pass and let me know what you
>> think. Overall, this is what I changed in the AIP:
>>
>> - Push based event-driven scheduling. I updated this section entirely
>> because I received many concerns about the previous proposal. The overall
>> idea now is to leverage the create asset event API endpoint to send
>> notifications from external (e.g. cloud provider) to Airflow environment.
>>
>> - I updated the poll based event-driven scheduling DAG author experience
>> to use a message queue scenario. I understood that this is probably the
>> main use case we are trying to cover with this AIP, thus I used it as
>> example and mentioned it multiple times across the AIP.
>>
>> Thanks again for your time :)
>>
>>
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
>>
>> Vincent
>>
>> On 2024/07/29 15:58:23 Vincent Beck wrote:
>> > Thanks a lot all for the comments, this is very much appreciated! I
>> received many comments from this thread and in confluence, thanks again.
>> I'll try to address them all in the AIP and will send an email in this
>> thread once done. I will most likely revisit the push-based approach given
>> the number of concerns I received, thanks Jarek for proposing another
>> solution, I'll probably go down that path.
>> >
>> > One follow-up question Vikram.
>> >
>> > > The bespoke triggerer approach completely makes sense for the long
>> tail here, but can we do better for the 20% of scenarios which cover well
>> over 80% of usage here is the question in my mind. Or, are you thinking of
>> those as being covered in the "push" model?
>> >
>> > Could you share more details about what is this "20% of scenarios which
>> cover well over 80% of usage" please?
>> >
>> > Vincent
>> >
>> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
>> > > Thanks Vincent for driving these, I have added my comments to the AIP
>> too.
>> > >
>> > > Regards,
>> > > Kaxil
>> > >
>> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
>> > > <Je...@de.bosch.com.invalid> wrote:
>> > >
>> > > > +1 on the comments of Vikram and Jarek, added main points on
>> confluence
>> > > >
>> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
>> > > > ________________________________
>> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
>> > > > Sent: Friday, July 26, 2024 8:46:55 PM
>> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
>> > > > Subject: Re: [DISCUSS] External event driven scheduling in Airflow
>> > > >
>> > > > Vincent,
>> > > >
>> > > > Thanks for writing this up. The overview looks really good!
>> > > >
>> > > > I will leave my comments in the AIP as well, but at a high level
>> they are
>> > > > both relatively focused on the "how", rather than the "what".
>> > > > With respect to the pull / polling approach, I completely agree
>> that some
>> > > > incarnation of this is needed.
>> > > > I am less certain as to how on this part. The bespoke triggerer
>> approach
>> > > > completely makes sense for the long tail here, but can we do better
>> for the
>> > > > 20% of scenarios which cover well over 80% of usage here is the
>> question in
>> > > > my mind. Or, are you thinking of those as being covered in the
>> "push"
>> > > > model?
>> > > >
>> > > > Which leads to the "push" model approach.
>> > > > I am struggling with the same question that Jarek raised here about
>> whether
>> > > > we need a new Airflow entity over and beyond the existing REST API
>> for the
>> > > > same.
>> > > > I am concerned about this becoming a vector of attack on Airflow.
>> > > > I see that this is a hot topic of discussion in the Confluence doc
>> as well,
>> > > > but wanted to summarize here as well, so it didn't get lost in the
>> threads
>> > > > of comments.
>> > > >
>> > > > Best regards,
>> > > > Vikram
>> > > >
>> > > >
>> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> > > >
>> > > > > Thanks Vincent. I took a look and I have a general comment. I
>> > > > > strongly think external driven scheduling is really needed -
>> especially,
>> > > > it
>> > > > > should be much easier for a user to "plug-in" such an external
>> event to
>> > > > > Airflow. And there are two parts of it - as correctly stated
>> there - pull
>> > > > > and push.
>> > > > >
>> > > > > For the pull - I think it would be great to have a kind of
>> specialized
>> > > > > Triggers that will be started when DAG is parsed - and those
>> Triggers
>> > > > could
>> > > > > generate the events for DAGs. I think basically that's all that is
>> > > > needed,
>> > > > > for example I imagine a pubsub trigger that will subscribe to
>> messages
>> > > > > coming on the pubsub queue and fire "Asset" event when a message
>> is
>> > > > > received. Not much controversy there - I am not sure about the
>> polling
>> > > > > thing , because I've always believed that when "asyncio-native"
>> Trigger
>> > > > is
>> > > > > run in the asyncio event loop, we do not "poll" every second or
>> so (but
>> > > > > maybe this is just coming from some specific triggers that
>> actually do
>> > > > > such regular poll. But yes - there are polls like running select
>> on the
>> > > > DB
>> > > > > that cannot be easily "async-ed" so having a configurable polling
>> time
>> > > > > would be good there (but I am not sure maybe it's even possible
>> today). I
>> > > > > think this would be really great if we have that option, because
>> it makes
>> > > > > it much easier to set up the authorization for Airlfow users -
>> rather
>> > > > than
>> > > > > setting up authorization and REST calls coming from an external
>> system,
>> > > > we
>> > > > > can utilize Connections of Airlfow to authorize such a Trigger to
>> > > > subscribe
>> > > > > to events.
>> > > > >
>> > > > > For the push proposal - as I read the proposal, the main point
>> behind it
>> > > > > is rather than users having to write "Airflow" way of triggering
>> events
>> > > > and
>> > > > > configuring authentication (using REST API) to generate asset
>> events, is
>> > > > to
>> > > > > make Airflow natively understand external ways of pushing - and
>> > > > effectively
>> > > > > authorizing and mapping such incoming unauthorized requests into
>> event
>> > > > that
>> > > > > could be generated by an API REST call.
>> > > > > I am not really sure honestly if this is something that we want as
>> > > > > "running" in airlfow as an endpoint. I'd say such an unauthorised
>> > > > endpoint
>> > > > > is probably not a good idea - for a variety of reasons, mostly
>> security.
>> > > > > And as I understand the goal is that users can easily point at
>> > > > "3rd-party"
>> > > > > notification to Airflow and get the event generated.
>> > > > >
>> > > > > My feeling is that while this is needed - it should be
>> externalised from
>> > > > > airlfow webserver. The authorization has to be set up anyway
>> > > > additionally -
>> > > > > unlike in "poll" case - we cannot use Connections for authorizing
>> > > > (because
>> > > > > it's not Airlfow that authorizes in an external system - it's the
>> other
>> > > > way
>> > > > > round). So we have to anyhow setup "something extra" in Airflow to
>> > > > > authorize the external system. Which could be what we have now -
>> user
>> > > > that
>> > > > > allows us to trigger the event. Which means that our REST API
>> could
>> > > > > potentially be used the same way it is now, but we will need
>> "something"
>> > > > > (library, lambda function etc.) that users could easily setup in
>> the
>> > > > > external system to map whatever trigger they generate natively
>> (say S3
>> > > > file
>> > > > > created) to Airflow REST API.
>> > > > >
>> > > > > As I see it - this is quite often used (and very practical, that
>> you
>> > > > deploy
>> > > > > a cloud function or lambda that subscribes on the event received
>> when
>> > > > > S3/GCS is created. So it would be on the user to deploy such a
>> lambda -
>> > > > but
>> > > > > we **could** provide a library of those: say s3 lambda, gcp cloud
>> > > > function
>> > > > > in respective providers - with documentation how to set them up,
>> and how
>> > > > to
>> > > > > configure authorization and we would be generally "done". I am
>> just not
>> > > > > sure if we need a new entity in Airflow for that (Event
>> receiver). It
>> > > > feels
>> > > > > like it asks Airflow to take more responsibility, when we all
>> think on
>> > > > what
>> > > > > to "remove" from Airflow rather than "add" to it - especially
>> when it
>> > > > comes
>> > > > > to external integrations. It feels to me that Airflow should make
>> it easy
>> > > > > to be triggered by such an external system and make it easy to
>> "map" to
>> > > > the
>> > > > > way we expect to get events triggered, but this should be done
>> outside of
>> > > > > Airflow. If the users can easily find in our docs when they
>> search "what
>> > > > do
>> > > > > I do to externally trigger Airflow on S3 change": either a)
>> configure
>> > > > > polling in airflow using s3 Connection, or b) "create a user +
>> deploy
>> > > > this
>> > > > > lambda with those parameters" - that is "easy enough" and very
>> practical
>> > > > > as well.
>> > > > >
>> > > > > But maybe I am not seeing the whole picture and the real problem
>> it's
>> > > > > solving - so take it as a "first review pass" and "guts feeling".
>> > > > >
>> > > > > J.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
>> > > > <vincbeck@amazon.com.invalid
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hello everyone,
>> > > > > >
>> > > > > > I created a draft AIP regarding "External event driven
>> scheduling in
>> > > > > > Airflow". This proposal is about adding capability in Airflow to
>> > > > schedule
>> > > > > > DAGs based on external events. Here are some examples of such
>> external
>> > > > > > events:
>> > > > > > - A user signs up to one of the user pool defined in my cloud
>> provider
>> > > > > > - One of the databases used in my company has been updated
>> > > > > > - A job in my cloud provider has been executed successfully
>> > > > > >
>> > > > > > The intent of this AIP is to leverage datasets (which will be
>> soon
>> > > > > assets)
>> > > > > > and update them based on external events. I would like to
>> propose this
>> > > > > AIP
>> > > > > > for discussion and more importantly, hear some feedbacks from
>> you :)
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
>> > > > <
>> > > >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
>> > > > >
>> > > > > >
>> > > > > > Vincent
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> For additional commands, e-mail: dev-help@airflow.apache.org
>>
>>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks Vincent, I will take a look again tomorrow.
On Tue, 30 Jul 2024 at 18:47, Vincent Beck <vi...@apache.org> wrote:
> Hi everyone,
>
> I updated the AIP-82 given the different comments and concerns I received.
> I also tried to reply to all comments individually. I would really
> appreciate if you can do a second pass and let me know what you think.
> Overall, this is what I changed in the AIP:
>
> - Push based event-driven scheduling. I updated this section entirely
> because I received many concerns about the previous proposal. The overall
> idea now is to leverage the create asset event API endpoint to send
> notifications from external (e.g. cloud provider) to Airflow environment.
>
> - I updated the poll based event-driven scheduling DAG author experience
> to use a message queue scenario. I understood that this is probably the
> main use case we are trying to cover with this AIP, thus I used it as
> example and mentioned it multiple times across the AIP.
>
> Thanks again for your time :)
>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
>
> Vincent
>
> On 2024/07/29 15:58:23 Vincent Beck wrote:
> > Thanks a lot all for the comments, this is very much appreciated! I
> received many comments from this thread and in confluence, thanks again.
> I'll try to address them all in the AIP and will send an email in this
> thread once done. I will most likely revisit the push-based approach given
> the number of concerns I received, thanks Jarek for proposing another
> solution, I'll probably go down that path.
> >
> > One follow-up question Vikram.
> >
> > > The bespoke triggerer approach completely makes sense for the long
> tail here, but can we do better for the 20% of scenarios which cover well
> over 80% of usage here is the question in my mind. Or, are you thinking of
> those as being covered in the "push" model?
> >
> > Could you share more details about what is this "20% of scenarios which
> cover well over 80% of usage" please?
> >
> > Vincent
> >
> > On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > > Thanks Vincent for driving these, I have added my comments to the AIP
> too.
> > >
> > > Regards,
> > > Kaxil
> > >
> > > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
> > > <Je...@de.bosch.com.invalid> wrote:
> > >
> > > > +1 on the comments of Vikram and Jarek, added main points on
> confluence
> > > >
> > > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > > ________________________________
> > > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > > Subject: Re: [DISCUSS] External event driven scheduling in Airflow
> > > >
> > > > Vincent,
> > > >
> > > > Thanks for writing this up. The overview looks really good!
> > > >
> > > > I will leave my comments in the AIP as well, but at a high level
> they are
> > > > both relatively focused on the "how", rather than the "what".
> > > > With respect to the pull / polling approach, I completely agree that
> some
> > > > incarnation of this is needed.
> > > > I am less certain as to how on this part. The bespoke triggerer
> approach
> > > > completely makes sense for the long tail here, but can we do better
> for the
> > > > 20% of scenarios which cover well over 80% of usage here is the
> question in
> > > > my mind. Or, are you thinking of those as being covered in the "push"
> > > > model?
> > > >
> > > > Which leads to the "push" model approach.
> > > > I am struggling with the same question that Jarek raised here about
> whether
> > > > we need a new Airflow entity over and beyond the existing REST API
> for the
> > > > same.
> > > > I am concerned about this becoming a vector of attack on Airflow.
> > > > I see that this is a hot topic of discussion in the Confluence doc
> as well,
> > > > but wanted to summarize here as well, so it didn't get lost in the
> threads
> > > > of comments.
> > > >
> > > > Best regards,
> > > > Vikram
> > > >
> > > >
> > > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > > > Thanks Vincent. I took a look and I have a general comment. I
> > > > > strongly think external driven scheduling is really needed -
> especially,
> > > > it
> > > > > should be much easier for a user to "plug-in" such an external
> event to
> > > > > Airflow. And there are two parts of it - as correctly stated there
> - pull
> > > > > and push.
> > > > >
> > > > > For the pull - I think it would be great to have a kind of
> specialized
> > > > > Triggers that will be started when DAG is parsed - and those
> Triggers
> > > > could
> > > > > generate the events for DAGs. I think basically that's all that is
> > > > needed,
> > > > > for example I imagine a pubsub trigger that will subscribe to
> messages
> > > > > coming on the pubsub queue and fire "Asset" event when a message is
> > > > > received. Not much controversy there - I am not sure about the
> polling
> > > > > thing , because I've always believed that when "asyncio-native"
> Trigger
> > > > is
> > > > > run in the asyncio event loop, we do not "poll" every second or so
> (but
> > > > > maybe this is just coming from some specific triggers that
> actually do
> > > > > such regular poll. But yes - there are polls like running select
> on the
> > > > DB
> > > > > that cannot be easily "async-ed" so having a configurable polling
> time
> > > > > would be good there (but I am not sure maybe it's even possible
> today). I
> > > > > think this would be really great if we have that option, because
> it makes
> > > > > it much easier to set up the authorization for Airlfow users -
> rather
> > > > than
> > > > > setting up authorization and REST calls coming from an external
> system,
> > > > we
> > > > > can utilize Connections of Airlfow to authorize such a Trigger to
> > > > subscribe
> > > > > to events.
> > > > >
> > > > > For the push proposal - as I read the proposal, the main point
> behind it
> > > > > is rather than users having to write "Airflow" way of triggering
> events
> > > > and
> > > > > configuring authentication (using REST API) to generate asset
> events, is
> > > > to
> > > > > make Airflow natively understand external ways of pushing - and
> > > > effectively
> > > > > authorizing and mapping such incoming unauthorized requests into
> event
> > > > that
> > > > > could be generated by an API REST call.
> > > > > I am not really sure honestly if this is something that we want as
> > > > > "running" in airlfow as an endpoint. I'd say such an unauthorised
> > > > endpoint
> > > > > is probably not a good idea - for a variety of reasons, mostly
> security.
> > > > > And as I understand the goal is that users can easily point at
> > > > "3rd-party"
> > > > > notification to Airflow and get the event generated.
> > > > >
> > > > > My feeling is that while this is needed - it should be
> externalised from
> > > > > airlfow webserver. The authorization has to be set up anyway
> > > > additionally -
> > > > > unlike in "poll" case - we cannot use Connections for authorizing
> > > > (because
> > > > > it's not Airlfow that authorizes in an external system - it's the
> other
> > > > way
> > > > > round). So we have to anyhow setup "something extra" in Airflow to
> > > > > authorize the external system. Which could be what we have now -
> user
> > > > that
> > > > > allows us to trigger the event. Which means that our REST API could
> > > > > potentially be used the same way it is now, but we will need
> "something"
> > > > > (library, lambda function etc.) that users could easily setup in
> the
> > > > > external system to map whatever trigger they generate natively
> (say S3
> > > > file
> > > > > created) to Airflow REST API.
> > > > >
> > > > > As I see it - this is quite often used (and very practical, that
> you
> > > > deploy
> > > > > a cloud function or lambda that subscribes on the event received
> when
> > > > > S3/GCS is created. So it would be on the user to deploy such a
> lambda -
> > > > but
> > > > > we **could** provide a library of those: say s3 lambda, gcp cloud
> > > > function
> > > > > in respective providers - with documentation how to set them up,
> and how
> > > > to
> > > > > configure authorization and we would be generally "done". I am
> just not
> > > > > sure if we need a new entity in Airflow for that (Event receiver).
> It
> > > > feels
> > > > > like it asks Airflow to take more responsibility, when we all
> think on
> > > > what
> > > > > to "remove" from Airflow rather than "add" to it - especially when
> it
> > > > comes
> > > > > to external integrations. It feels to me that Airflow should make
> it easy
> > > > > to be triggered by such an external system and make it easy to
> "map" to
> > > > the
> > > > > way we expect to get events triggered, but this should be done
> outside of
> > > > > Airflow. If the users can easily find in our docs when they search
> "what
> > > > do
> > > > > I do to externally trigger Airflow on S3 change": either a)
> configure
> > > > > polling in airflow using s3 Connection, or b) "create a user +
> deploy
> > > > this
> > > > > lambda with those parameters" - that is "easy enough" and very
> practical
> > > > > as well.
> > > > >
> > > > > But maybe I am not seeing the whole picture and the real problem
> it's
> > > > > solving - so take it as a "first review pass" and "guts feeling".
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > > <vincbeck@amazon.com.invalid
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I created a draft AIP regarding "External event driven
> scheduling in
> > > > > > Airflow". This proposal is about adding capability in Airflow to
> > > > schedule
> > > > > > DAGs based on external events. Here are some examples of such
> external
> > > > > > events:
> > > > > > - A user signs up to one of the user pool defined in my cloud
> provider
> > > > > > - One of the databases used in my company has been updated
> > > > > > - A job in my cloud provider has been executed successfully
> > > > > >
> > > > > > The intent of this AIP is to leverage datasets (which will be
> soon
> > > > > assets)
> > > > > > and update them based on external events. I would like to
> propose this
> > > > > AIP
> > > > > > for discussion and more importantly, hear some feedbacks from
> you :)
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > > <
> > > >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > > >
> > > > > >
> > > > > > Vincent
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vincent Beck <vi...@apache.org>.
Hi everyone,
I updated the AIP-82 given the different comments and concerns I received. I also tried to reply to all comments individually. I would really appreciate if you can do a second pass and let me know what you think. Overall, this is what I changed in the AIP:
- Push based event-driven scheduling. I updated this section entirely because I received many concerns about the previous proposal. The overall idea now is to leverage the create asset event API endpoint to send notifications from external (e.g. cloud provider) to Airflow environment.
- I updated the poll based event-driven scheduling DAG author experience to use a message queue scenario. I understood that this is probably the main use case we are trying to cover with this AIP, thus I used it as example and mentioned it multiple times across the AIP.
Thanks again for your time :)
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
Vincent
On 2024/07/29 15:58:23 Vincent Beck wrote:
> Thanks a lot all for the comments, this is very much appreciated! I received many comments from this thread and in confluence, thanks again. I'll try to address them all in the AIP and will send an email in this thread once done. I will most likely revisit the push-based approach given the number of concerns I received, thanks Jarek for proposing another solution, I'll probably go down that path.
>
> One follow-up question Vikram.
>
> > The bespoke triggerer approach completely makes sense for the long tail here, but can we do better for the 20% of scenarios which cover well over 80% of usage here is the question in my mind. Or, are you thinking of those as being covered in the "push" model?
>
> Could you share more details about what is this "20% of scenarios which cover well over 80% of usage" please?
>
> Vincent
>
> On 2024/07/29 15:37:50 Kaxil Naik wrote:
> > Thanks Vincent for driving these, I have added my comments to the AIP too.
> >
> > Regards,
> > Kaxil
> >
> > On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
> > <Je...@de.bosch.com.invalid> wrote:
> >
> > > +1 on the comments of Vikram and Jarek, added main points on confluence
> > >
> > > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > > ________________________________
> > > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > > Sent: Friday, July 26, 2024 8:46:55 PM
> > > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > > Subject: Re: [DISCUSS] External event driven scheduling in Airflow
> > >
> > > Vincent,
> > >
> > > Thanks for writing this up. The overview looks really good!
> > >
> > > I will leave my comments in the AIP as well, but at a high level they are
> > > both relatively focused on the "how", rather than the "what".
> > > With respect to the pull / polling approach, I completely agree that some
> > > incarnation of this is needed.
> > > I am less certain as to how on this part. The bespoke triggerer approach
> > > completely makes sense for the long tail here, but can we do better for the
> > > 20% of scenarios which cover well over 80% of usage here is the question in
> > > my mind. Or, are you thinking of those as being covered in the "push"
> > > model?
> > >
> > > Which leads to the "push" model approach.
> > > I am struggling with the same question that Jarek raised here about whether
> > > we need a new Airflow entity over and beyond the existing REST API for the
> > > same.
> > > I am concerned about this becoming a vector of attack on Airflow.
> > > I see that this is a hot topic of discussion in the Confluence doc as well,
> > > but wanted to summarize here as well, so it didn't get lost in the threads
> > > of comments.
> > >
> > > Best regards,
> > > Vikram
> > >
> > >
> > > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > Thanks Vincent. I took a look and I have a general comment. I
> > > > strongly think external driven scheduling is really needed - especially,
> > > it
> > > > should be much easier for a user to "plug-in" such an external event to
> > > > Airflow. And there are two parts of it - as correctly stated there - pull
> > > > and push.
> > > >
> > > > For the pull - I think it would be great to have a kind of specialized
> > > > Triggers that will be started when DAG is parsed - and those Triggers
> > > could
> > > > generate the events for DAGs. I think basically that's all that is
> > > needed,
> > > > for example I imagine a pubsub trigger that will subscribe to messages
> > > > coming on the pubsub queue and fire "Asset" event when a message is
> > > > received. Not much controversy there - I am not sure about the polling
> > > > thing , because I've always believed that when "asyncio-native" Trigger
> > > is
> > > > run in the asyncio event loop, we do not "poll" every second or so (but
> > > > maybe this is just coming from some specific triggers that actually do
> > > > such regular poll. But yes - there are polls like running select on the
> > > DB
> > > > that cannot be easily "async-ed" so having a configurable polling time
> > > > would be good there (but I am not sure maybe it's even possible today). I
> > > > think this would be really great if we have that option, because it makes
> > > > it much easier to set up the authorization for Airlfow users - rather
> > > than
> > > > setting up authorization and REST calls coming from an external system,
> > > we
> > > > can utilize Connections of Airlfow to authorize such a Trigger to
> > > subscribe
> > > > to events.
> > > >
> > > > For the push proposal - as I read the proposal, the main point behind it
> > > > is rather than users having to write "Airflow" way of triggering events
> > > and
> > > > configuring authentication (using REST API) to generate asset events, is
> > > to
> > > > make Airflow natively understand external ways of pushing - and
> > > effectively
> > > > authorizing and mapping such incoming unauthorized requests into event
> > > that
> > > > could be generated by an API REST call.
> > > > I am not really sure honestly if this is something that we want as
> > > > "running" in airlfow as an endpoint. I'd say such an unauthorised
> > > endpoint
> > > > is probably not a good idea - for a variety of reasons, mostly security.
> > > > And as I understand the goal is that users can easily point at
> > > "3rd-party"
> > > > notification to Airflow and get the event generated.
> > > >
> > > > My feeling is that while this is needed - it should be externalised from
> > > > airlfow webserver. The authorization has to be set up anyway
> > > additionally -
> > > > unlike in "poll" case - we cannot use Connections for authorizing
> > > (because
> > > > it's not Airlfow that authorizes in an external system - it's the other
> > > way
> > > > round). So we have to anyhow setup "something extra" in Airflow to
> > > > authorize the external system. Which could be what we have now - user
> > > that
> > > > allows us to trigger the event. Which means that our REST API could
> > > > potentially be used the same way it is now, but we will need "something"
> > > > (library, lambda function etc.) that users could easily setup in the
> > > > external system to map whatever trigger they generate natively (say S3
> > > file
> > > > created) to Airflow REST API.
> > > >
> > > > As I see it - this is quite often used (and very practical, that you
> > > deploy
> > > > a cloud function or lambda that subscribes on the event received when
> > > > S3/GCS is created. So it would be on the user to deploy such a lambda -
> > > but
> > > > we **could** provide a library of those: say s3 lambda, gcp cloud
> > > function
> > > > in respective providers - with documentation how to set them up, and how
> > > to
> > > > configure authorization and we would be generally "done". I am just not
> > > > sure if we need a new entity in Airflow for that (Event receiver). It
> > > feels
> > > > like it asks Airflow to take more responsibility, when we all think on
> > > what
> > > > to "remove" from Airflow rather than "add" to it - especially when it
> > > comes
> > > > to external integrations. It feels to me that Airflow should make it easy
> > > > to be triggered by such an external system and make it easy to "map" to
> > > the
> > > > way we expect to get events triggered, but this should be done outside of
> > > > Airflow. If the users can easily find in our docs when they search "what
> > > do
> > > > I do to externally trigger Airflow on S3 change": either a) configure
> > > > polling in airflow using s3 Connection, or b) "create a user + deploy
> > > this
> > > > lambda with those parameters" - that is "easy enough" and very practical
> > > > as well.
> > > >
> > > > But maybe I am not seeing the whole picture and the real problem it's
> > > > solving - so take it as a "first review pass" and "guts feeling".
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > > <vincbeck@amazon.com.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I created a draft AIP regarding "External event driven scheduling in
> > > > > Airflow". This proposal is about adding capability in Airflow to
> > > schedule
> > > > > DAGs based on external events. Here are some examples of such external
> > > > > events:
> > > > > - A user signs up to one of the user pool defined in my cloud provider
> > > > > - One of the databases used in my company has been updated
> > > > > - A job in my cloud provider has been executed successfully
> > > > >
> > > > > The intent of this AIP is to leverage datasets (which will be soon
> > > > assets)
> > > > > and update them based on external events. I would like to propose this
> > > > AIP
> > > > > for discussion and more importantly, hear some feedbacks from you :)
> > > > >
> > > > >
> > > > >
> > > >
> > > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > > <
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > > >
> > > > >
> > > > > Vincent
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vincent Beck <vi...@apache.org>.
Thanks a lot all for the comments, this is very much appreciated! I received many comments from this thread and in confluence, thanks again. I'll try to address them all in the AIP and will send an email in this thread once done. I will most likely revisit the push-based approach given the number of concerns I received, thanks Jarek for proposing another solution, I'll probably go down that path.
One follow-up question Vikram.
> The bespoke triggerer approach completely makes sense for the long tail here, but can we do better for the 20% of scenarios which cover well over 80% of usage here is the question in my mind. Or, are you thinking of those as being covered in the "push" model?
Could you share more details about what is this "20% of scenarios which cover well over 80% of usage" please?
Vincent
On 2024/07/29 15:37:50 Kaxil Naik wrote:
> Thanks Vincent for driving these, I have added my comments to the AIP too.
>
> Regards,
> Kaxil
>
> On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
> <Je...@de.bosch.com.invalid> wrote:
>
> > +1 on the comments of Vikram and Jarek, added main points on confluence
> >
> > Sent from Outlook for iOS<https://aka.ms/o0ukef>
> > ________________________________
> > From: Vikram Koka <vi...@astronomer.io.INVALID>
> > Sent: Friday, July 26, 2024 8:46:55 PM
> > To: dev@airflow.apache.org <de...@airflow.apache.org>
> > Subject: Re: [DISCUSS] External event driven scheduling in Airflow
> >
> > Vincent,
> >
> > Thanks for writing this up. The overview looks really good!
> >
> > I will leave my comments in the AIP as well, but at a high level they are
> > both relatively focused on the "how", rather than the "what".
> > With respect to the pull / polling approach, I completely agree that some
> > incarnation of this is needed.
> > I am less certain as to how on this part. The bespoke triggerer approach
> > completely makes sense for the long tail here, but can we do better for the
> > 20% of scenarios which cover well over 80% of usage here is the question in
> > my mind. Or, are you thinking of those as being covered in the "push"
> > model?
> >
> > Which leads to the "push" model approach.
> > I am struggling with the same question that Jarek raised here about whether
> > we need a new Airflow entity over and beyond the existing REST API for the
> > same.
> > I am concerned about this becoming a vector of attack on Airflow.
> > I see that this is a hot topic of discussion in the Confluence doc as well,
> > but wanted to summarize here as well, so it didn't get lost in the threads
> > of comments.
> >
> > Best regards,
> > Vikram
> >
> >
> > On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Thanks Vincent. I took a look and I have a general comment. I
> > > strongly think external driven scheduling is really needed - especially,
> > it
> > > should be much easier for a user to "plug-in" such an external event to
> > > Airflow. And there are two parts of it - as correctly stated there - pull
> > > and push.
> > >
> > > For the pull - I think it would be great to have a kind of specialized
> > > Triggers that will be started when DAG is parsed - and those Triggers
> > could
> > > generate the events for DAGs. I think basically that's all that is
> > needed,
> > > for example I imagine a pubsub trigger that will subscribe to messages
> > > coming on the pubsub queue and fire "Asset" event when a message is
> > > received. Not much controversy there - I am not sure about the polling
> > > thing , because I've always believed that when "asyncio-native" Trigger
> > is
> > > run in the asyncio event loop, we do not "poll" every second or so (but
> > > maybe this is just coming from some specific triggers that actually do
> > > such regular poll. But yes - there are polls like running select on the
> > DB
> > > that cannot be easily "async-ed" so having a configurable polling time
> > > would be good there (but I am not sure maybe it's even possible today). I
> > > think this would be really great if we have that option, because it makes
> > > it much easier to set up the authorization for Airlfow users - rather
> > than
> > > setting up authorization and REST calls coming from an external system,
> > we
> > > can utilize Connections of Airlfow to authorize such a Trigger to
> > subscribe
> > > to events.
> > >
> > > For the push proposal - as I read the proposal, the main point behind it
> > > is rather than users having to write "Airflow" way of triggering events
> > and
> > > configuring authentication (using REST API) to generate asset events, is
> > to
> > > make Airflow natively understand external ways of pushing - and
> > effectively
> > > authorizing and mapping such incoming unauthorized requests into event
> > that
> > > could be generated by an API REST call.
> > > I am not really sure honestly if this is something that we want as
> > > "running" in airlfow as an endpoint. I'd say such an unauthorised
> > endpoint
> > > is probably not a good idea - for a variety of reasons, mostly security.
> > > And as I understand the goal is that users can easily point at
> > "3rd-party"
> > > notification to Airflow and get the event generated.
> > >
> > > My feeling is that while this is needed - it should be externalised from
> > > airlfow webserver. The authorization has to be set up anyway
> > additionally -
> > > unlike in "poll" case - we cannot use Connections for authorizing
> > (because
> > > it's not Airlfow that authorizes in an external system - it's the other
> > way
> > > round). So we have to anyhow setup "something extra" in Airflow to
> > > authorize the external system. Which could be what we have now - user
> > that
> > > allows us to trigger the event. Which means that our REST API could
> > > potentially be used the same way it is now, but we will need "something"
> > > (library, lambda function etc.) that users could easily setup in the
> > > external system to map whatever trigger they generate natively (say S3
> > file
> > > created) to Airflow REST API.
> > >
> > > As I see it - this is quite often used (and very practical, that you
> > deploy
> > > a cloud function or lambda that subscribes on the event received when
> > > S3/GCS is created. So it would be on the user to deploy such a lambda -
> > but
> > > we **could** provide a library of those: say s3 lambda, gcp cloud
> > function
> > > in respective providers - with documentation how to set them up, and how
> > to
> > > configure authorization and we would be generally "done". I am just not
> > > sure if we need a new entity in Airflow for that (Event receiver). It
> > feels
> > > like it asks Airflow to take more responsibility, when we all think on
> > what
> > > to "remove" from Airflow rather than "add" to it - especially when it
> > comes
> > > to external integrations. It feels to me that Airflow should make it easy
> > > to be triggered by such an external system and make it easy to "map" to
> > the
> > > way we expect to get events triggered, but this should be done outside of
> > > Airflow. If the users can easily find in our docs when they search "what
> > do
> > > I do to externally trigger Airflow on S3 change": either a) configure
> > > polling in airflow using s3 Connection, or b) "create a user + deploy
> > this
> > > lambda with those parameters" - that is "easy enough" and very practical
> > > as well.
> > >
> > > But maybe I am not seeing the whole picture and the real problem it's
> > > solving - so take it as a "first review pass" and "guts feeling".
> > >
> > > J.
> > >
> > >
> > >
> > >
> > > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> > <vincbeck@amazon.com.invalid
> > > >
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I created a draft AIP regarding "External event driven scheduling in
> > > > Airflow". This proposal is about adding capability in Airflow to
> > schedule
> > > > DAGs based on external events. Here are some examples of such external
> > > > events:
> > > > - A user signs up to one of the user pool defined in my cloud provider
> > > > - One of the databases used in my company has been updated
> > > > - A job in my cloud provider has been executed successfully
> > > >
> > > > The intent of this AIP is to leverage datasets (which will be soon
> > > assets)
> > > > and update them based on external events. I would like to propose this
> > > AIP
> > > > for discussion and more importantly, hear some feedbacks from you :)
> > > >
> > > >
> > > >
> > >
> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> > <
> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> > >
> > > >
> > > > Vincent
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks Vincent for driving these, I have added my comments to the AIP too.
Regards,
Kaxil
On Fri, 26 Jul 2024 at 20:16, Scheffler Jens (XC-AS/EAE-ADA-T)
<Je...@de.bosch.com.invalid> wrote:
> +1 on the comments of Vikram and Jarek, added main points on confluence
>
> Sent from Outlook for iOS<https://aka.ms/o0ukef>
> ________________________________
> From: Vikram Koka <vi...@astronomer.io.INVALID>
> Sent: Friday, July 26, 2024 8:46:55 PM
> To: dev@airflow.apache.org <de...@airflow.apache.org>
> Subject: Re: [DISCUSS] External event driven scheduling in Airflow
>
> Vincent,
>
> Thanks for writing this up. The overview looks really good!
>
> I will leave my comments in the AIP as well, but at a high level they are
> both relatively focused on the "how", rather than the "what".
> With respect to the pull / polling approach, I completely agree that some
> incarnation of this is needed.
> I am less certain as to how on this part. The bespoke triggerer approach
> completely makes sense for the long tail here, but can we do better for the
> 20% of scenarios which cover well over 80% of usage here is the question in
> my mind. Or, are you thinking of those as being covered in the "push"
> model?
>
> Which leads to the "push" model approach.
> I am struggling with the same question that Jarek raised here about whether
> we need a new Airflow entity over and beyond the existing REST API for the
> same.
> I am concerned about this becoming a vector of attack on Airflow.
> I see that this is a hot topic of discussion in the Confluence doc as well,
> but wanted to summarize here as well, so it didn't get lost in the threads
> of comments.
>
> Best regards,
> Vikram
>
>
> On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Thanks Vincent. I took a look and I have a general comment. I
> > strongly think external driven scheduling is really needed - especially,
> it
> > should be much easier for a user to "plug-in" such an external event to
> > Airflow. And there are two parts of it - as correctly stated there - pull
> > and push.
> >
> > For the pull - I think it would be great to have a kind of specialized
> > Triggers that will be started when DAG is parsed - and those Triggers
> could
> > generate the events for DAGs. I think basically that's all that is
> needed,
> > for example I imagine a pubsub trigger that will subscribe to messages
> > coming on the pubsub queue and fire "Asset" event when a message is
> > received. Not much controversy there - I am not sure about the polling
> > thing , because I've always believed that when "asyncio-native" Trigger
> is
> > run in the asyncio event loop, we do not "poll" every second or so (but
> > maybe this is just coming from some specific triggers that actually do
> > such regular poll. But yes - there are polls like running select on the
> DB
> > that cannot be easily "async-ed" so having a configurable polling time
> > would be good there (but I am not sure maybe it's even possible today). I
> > think this would be really great if we have that option, because it makes
> > it much easier to set up the authorization for Airlfow users - rather
> than
> > setting up authorization and REST calls coming from an external system,
> we
> > can utilize Connections of Airlfow to authorize such a Trigger to
> subscribe
> > to events.
> >
> > For the push proposal - as I read the proposal, the main point behind it
> > is rather than users having to write "Airflow" way of triggering events
> and
> > configuring authentication (using REST API) to generate asset events, is
> to
> > make Airflow natively understand external ways of pushing - and
> effectively
> > authorizing and mapping such incoming unauthorized requests into event
> that
> > could be generated by an API REST call.
> > I am not really sure honestly if this is something that we want as
> > "running" in airlfow as an endpoint. I'd say such an unauthorised
> endpoint
> > is probably not a good idea - for a variety of reasons, mostly security.
> > And as I understand the goal is that users can easily point at
> "3rd-party"
> > notification to Airflow and get the event generated.
> >
> > My feeling is that while this is needed - it should be externalised from
> > airlfow webserver. The authorization has to be set up anyway
> additionally -
> > unlike in "poll" case - we cannot use Connections for authorizing
> (because
> > it's not Airlfow that authorizes in an external system - it's the other
> way
> > round). So we have to anyhow setup "something extra" in Airflow to
> > authorize the external system. Which could be what we have now - user
> that
> > allows us to trigger the event. Which means that our REST API could
> > potentially be used the same way it is now, but we will need "something"
> > (library, lambda function etc.) that users could easily setup in the
> > external system to map whatever trigger they generate natively (say S3
> file
> > created) to Airflow REST API.
> >
> > As I see it - this is quite often used (and very practical, that you
> deploy
> > a cloud function or lambda that subscribes on the event received when
> > S3/GCS is created. So it would be on the user to deploy such a lambda -
> but
> > we **could** provide a library of those: say s3 lambda, gcp cloud
> function
> > in respective providers - with documentation how to set them up, and how
> to
> > configure authorization and we would be generally "done". I am just not
> > sure if we need a new entity in Airflow for that (Event receiver). It
> feels
> > like it asks Airflow to take more responsibility, when we all think on
> what
> > to "remove" from Airflow rather than "add" to it - especially when it
> comes
> > to external integrations. It feels to me that Airflow should make it easy
> > to be triggered by such an external system and make it easy to "map" to
> the
> > way we expect to get events triggered, but this should be done outside of
> > Airflow. If the users can easily find in our docs when they search "what
> do
> > I do to externally trigger Airflow on S3 change": either a) configure
> > polling in airflow using s3 Connection, or b) "create a user + deploy
> this
> > lambda with those parameters" - that is "easy enough" and very practical
> > as well.
> >
> > But maybe I am not seeing the whole picture and the real problem it's
> > solving - so take it as a "first review pass" and "guts feeling".
> >
> > J.
> >
> >
> >
> >
> > On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent
> <vincbeck@amazon.com.invalid
> > >
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I created a draft AIP regarding "External event driven scheduling in
> > > Airflow". This proposal is about adding capability in Airflow to
> schedule
> > > DAGs based on external events. Here are some examples of such external
> > > events:
> > > - A user signs up to one of the user pool defined in my cloud provider
> > > - One of the databases used in my company has been updated
> > > - A job in my cloud provider has been executed successfully
> > >
> > > The intent of this AIP is to leverage datasets (which will be soon
> > assets)
> > > and update them based on external events. I would like to propose this
> > AIP
> > > for discussion and more importantly, hear some feedbacks from you :)
> > >
> > >
> > >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0
> <
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> >
> > >
> > > Vincent
> > >
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by "Scheffler Jens (XC-AS/EAE-ADA-T)" <Je...@de.bosch.com.INVALID>.
+1 on the comments of Vikram and Jarek, added main points on confluence
Sent from Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Vikram Koka <vi...@astronomer.io.INVALID>
Sent: Friday, July 26, 2024 8:46:55 PM
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: Re: [DISCUSS] External event driven scheduling in Airflow
Vincent,
Thanks for writing this up. The overview looks really good!
I will leave my comments in the AIP as well, but at a high level they are
both relatively focused on the "how", rather than the "what".
With respect to the pull / polling approach, I completely agree that some
incarnation of this is needed.
I am less certain as to how on this part. The bespoke triggerer approach
completely makes sense for the long tail here, but can we do better for the
20% of scenarios which cover well over 80% of usage here is the question in
my mind. Or, are you thinking of those as being covered in the "push"
model?
Which leads to the "push" model approach.
I am struggling with the same question that Jarek raised here about whether
we need a new Airflow entity over and beyond the existing REST API for the
same.
I am concerned about this becoming a vector of attack on Airflow.
I see that this is a hot topic of discussion in the Confluence doc as well,
but wanted to summarize here as well, so it didn't get lost in the threads
of comments.
Best regards,
Vikram
On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Thanks Vincent. I took a look and I have a general comment. I
> strongly think external driven scheduling is really needed - especially, it
> should be much easier for a user to "plug-in" such an external event to
> Airflow. And there are two parts of it - as correctly stated there - pull
> and push.
>
> For the pull - I think it would be great to have a kind of specialized
> Triggers that will be started when DAG is parsed - and those Triggers could
> generate the events for DAGs. I think basically that's all that is needed,
> for example I imagine a pubsub trigger that will subscribe to messages
> coming on the pubsub queue and fire "Asset" event when a message is
> received. Not much controversy there - I am not sure about the polling
> thing , because I've always believed that when "asyncio-native" Trigger is
> run in the asyncio event loop, we do not "poll" every second or so (but
> maybe this is just coming from some specific triggers that actually do
> such regular poll. But yes - there are polls like running select on the DB
> that cannot be easily "async-ed" so having a configurable polling time
> would be good there (but I am not sure maybe it's even possible today). I
> think this would be really great if we have that option, because it makes
> it much easier to set up the authorization for Airlfow users - rather than
> setting up authorization and REST calls coming from an external system, we
> can utilize Connections of Airlfow to authorize such a Trigger to subscribe
> to events.
>
> For the push proposal - as I read the proposal, the main point behind it
> is rather than users having to write "Airflow" way of triggering events and
> configuring authentication (using REST API) to generate asset events, is to
> make Airflow natively understand external ways of pushing - and effectively
> authorizing and mapping such incoming unauthorized requests into event that
> could be generated by an API REST call.
> I am not really sure honestly if this is something that we want as
> "running" in airlfow as an endpoint. I'd say such an unauthorised endpoint
> is probably not a good idea - for a variety of reasons, mostly security.
> And as I understand the goal is that users can easily point at "3rd-party"
> notification to Airflow and get the event generated.
>
> My feeling is that while this is needed - it should be externalised from
> airlfow webserver. The authorization has to be set up anyway additionally -
> unlike in "poll" case - we cannot use Connections for authorizing (because
> it's not Airlfow that authorizes in an external system - it's the other way
> round). So we have to anyhow setup "something extra" in Airflow to
> authorize the external system. Which could be what we have now - user that
> allows us to trigger the event. Which means that our REST API could
> potentially be used the same way it is now, but we will need "something"
> (library, lambda function etc.) that users could easily setup in the
> external system to map whatever trigger they generate natively (say S3 file
> created) to Airflow REST API.
>
> As I see it - this is quite often used (and very practical, that you deploy
> a cloud function or lambda that subscribes on the event received when
> S3/GCS is created. So it would be on the user to deploy such a lambda - but
> we **could** provide a library of those: say s3 lambda, gcp cloud function
> in respective providers - with documentation how to set them up, and how to
> configure authorization and we would be generally "done". I am just not
> sure if we need a new entity in Airflow for that (Event receiver). It feels
> like it asks Airflow to take more responsibility, when we all think on what
> to "remove" from Airflow rather than "add" to it - especially when it comes
> to external integrations. It feels to me that Airflow should make it easy
> to be triggered by such an external system and make it easy to "map" to the
> way we expect to get events triggered, but this should be done outside of
> Airflow. If the users can easily find in our docs when they search "what do
> I do to externally trigger Airflow on S3 change": either a) configure
> polling in airflow using s3 Connection, or b) "create a user + deploy this
> lambda with those parameters" - that is "easy enough" and very practical
> as well.
>
> But maybe I am not seeing the whole picture and the real problem it's
> solving - so take it as a "first review pass" and "guts feeling".
>
> J.
>
>
>
>
> On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent <vincbeck@amazon.com.invalid
> >
> wrote:
>
> > Hello everyone,
> >
> > I created a draft AIP regarding "External event driven scheduling in
> > Airflow". This proposal is about adding capability in Airflow to schedule
> > DAGs based on external events. Here are some examples of such external
> > events:
> > - A user signs up to one of the user pool defined in my cloud provider
> > - One of the databases used in my company has been updated
> > - A job in my cloud provider has been executed successfully
> >
> > The intent of this AIP is to leverage datasets (which will be soon
> assets)
> > and update them based on external events. I would like to propose this
> AIP
> > for discussion and more importantly, hear some feedbacks from you :)
> >
> >
> >
> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAIRFLOW%2FAIP-82%2BExternal%2Bevent%2Bdriven%2Bscheduling%2Bin%2BAirflow&data=05%7C02%7CJens.Scheffler%40de.bosch.com%7C9e55ef9af31e4a669ef108dcada3a726%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C638576165598178951%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3FFvhCI6RA6sPhZoiOBAqzgyTkC6NNYqJYjBRVqEmUY%3D&reserved=0<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow>
> >
> > Vincent
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Vikram Koka <vi...@astronomer.io.INVALID>.
Vincent,
Thanks for writing this up. The overview looks really good!
I will leave my comments in the AIP as well, but at a high level they are
both relatively focused on the "how", rather than the "what".
With respect to the pull / polling approach, I completely agree that some
incarnation of this is needed.
I am less certain as to how on this part. The bespoke triggerer approach
completely makes sense for the long tail here, but can we do better for the
20% of scenarios which cover well over 80% of usage here is the question in
my mind. Or, are you thinking of those as being covered in the "push"
model?
Which leads to the "push" model approach.
I am struggling with the same question that Jarek raised here about whether
we need a new Airflow entity over and beyond the existing REST API for the
same.
I am concerned about this becoming a vector of attack on Airflow.
I see that this is a hot topic of discussion in the Confluence doc as well,
but wanted to summarize here as well, so it didn't get lost in the threads
of comments.
Best regards,
Vikram
On Fri, Jul 26, 2024 at 5:16 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Thanks Vincent. I took a look and I have a general comment. I
> strongly think external driven scheduling is really needed - especially, it
> should be much easier for a user to "plug-in" such an external event to
> Airflow. And there are two parts of it - as correctly stated there - pull
> and push.
>
> For the pull - I think it would be great to have a kind of specialized
> Triggers that will be started when DAG is parsed - and those Triggers could
> generate the events for DAGs. I think basically that's all that is needed,
> for example I imagine a pubsub trigger that will subscribe to messages
> coming on the pubsub queue and fire "Asset" event when a message is
> received. Not much controversy there - I am not sure about the polling
> thing , because I've always believed that when "asyncio-native" Trigger is
> run in the asyncio event loop, we do not "poll" every second or so (but
> maybe this is just coming from some specific triggers that actually do
> such regular poll. But yes - there are polls like running select on the DB
> that cannot be easily "async-ed" so having a configurable polling time
> would be good there (but I am not sure maybe it's even possible today). I
> think this would be really great if we have that option, because it makes
> it much easier to set up the authorization for Airlfow users - rather than
> setting up authorization and REST calls coming from an external system, we
> can utilize Connections of Airlfow to authorize such a Trigger to subscribe
> to events.
>
> For the push proposal - as I read the proposal, the main point behind it
> is rather than users having to write "Airflow" way of triggering events and
> configuring authentication (using REST API) to generate asset events, is to
> make Airflow natively understand external ways of pushing - and effectively
> authorizing and mapping such incoming unauthorized requests into event that
> could be generated by an API REST call.
> I am not really sure honestly if this is something that we want as
> "running" in airlfow as an endpoint. I'd say such an unauthorised endpoint
> is probably not a good idea - for a variety of reasons, mostly security.
> And as I understand the goal is that users can easily point at "3rd-party"
> notification to Airflow and get the event generated.
>
> My feeling is that while this is needed - it should be externalised from
> airlfow webserver. The authorization has to be set up anyway additionally -
> unlike in "poll" case - we cannot use Connections for authorizing (because
> it's not Airlfow that authorizes in an external system - it's the other way
> round). So we have to anyhow setup "something extra" in Airflow to
> authorize the external system. Which could be what we have now - user that
> allows us to trigger the event. Which means that our REST API could
> potentially be used the same way it is now, but we will need "something"
> (library, lambda function etc.) that users could easily setup in the
> external system to map whatever trigger they generate natively (say S3 file
> created) to Airflow REST API.
>
> As I see it - this is quite often used (and very practical, that you deploy
> a cloud function or lambda that subscribes on the event received when
> S3/GCS is created. So it would be on the user to deploy such a lambda - but
> we **could** provide a library of those: say s3 lambda, gcp cloud function
> in respective providers - with documentation how to set them up, and how to
> configure authorization and we would be generally "done". I am just not
> sure if we need a new entity in Airflow for that (Event receiver). It feels
> like it asks Airflow to take more responsibility, when we all think on what
> to "remove" from Airflow rather than "add" to it - especially when it comes
> to external integrations. It feels to me that Airflow should make it easy
> to be triggered by such an external system and make it easy to "map" to the
> way we expect to get events triggered, but this should be done outside of
> Airflow. If the users can easily find in our docs when they search "what do
> I do to externally trigger Airflow on S3 change": either a) configure
> polling in airflow using s3 Connection, or b) "create a user + deploy this
> lambda with those parameters" - that is "easy enough" and very practical
> as well.
>
> But maybe I am not seeing the whole picture and the real problem it's
> solving - so take it as a "first review pass" and "guts feeling".
>
> J.
>
>
>
>
> On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent <vincbeck@amazon.com.invalid
> >
> wrote:
>
> > Hello everyone,
> >
> > I created a draft AIP regarding "External event driven scheduling in
> > Airflow". This proposal is about adding capability in Airflow to schedule
> > DAGs based on external events. Here are some examples of such external
> > events:
> > - A user signs up to one of the user pool defined in my cloud provider
> > - One of the databases used in my company has been updated
> > - A job in my cloud provider has been executed successfully
> >
> > The intent of this AIP is to leverage datasets (which will be soon
> assets)
> > and update them based on external events. I would like to propose this
> AIP
> > for discussion and more importantly, hear some feedbacks from you :)
> >
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
> >
> > Vincent
> >
>
Re: [DISCUSS] External event driven scheduling in Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
Thanks Vincent. I took a look and I have a general comment. I
strongly think external driven scheduling is really needed - especially, it
should be much easier for a user to "plug-in" such an external event to
Airflow. And there are two parts of it - as correctly stated there - pull
and push.
For the pull - I think it would be great to have a kind of specialized
Triggers that will be started when DAG is parsed - and those Triggers could
generate the events for DAGs. I think basically that's all that is needed,
for example I imagine a pubsub trigger that will subscribe to messages
coming on the pubsub queue and fire "Asset" event when a message is
received. Not much controversy there - I am not sure about the polling
thing , because I've always believed that when "asyncio-native" Trigger is
run in the asyncio event loop, we do not "poll" every second or so (but
maybe this is just coming from some specific triggers that actually do
such regular poll. But yes - there are polls like running select on the DB
that cannot be easily "async-ed" so having a configurable polling time
would be good there (but I am not sure maybe it's even possible today). I
think this would be really great if we have that option, because it makes
it much easier to set up the authorization for Airlfow users - rather than
setting up authorization and REST calls coming from an external system, we
can utilize Connections of Airlfow to authorize such a Trigger to subscribe
to events.
For the push proposal - as I read the proposal, the main point behind it
is rather than users having to write "Airflow" way of triggering events and
configuring authentication (using REST API) to generate asset events, is to
make Airflow natively understand external ways of pushing - and effectively
authorizing and mapping such incoming unauthorized requests into event that
could be generated by an API REST call.
I am not really sure honestly if this is something that we want as
"running" in airlfow as an endpoint. I'd say such an unauthorised endpoint
is probably not a good idea - for a variety of reasons, mostly security.
And as I understand the goal is that users can easily point at "3rd-party"
notification to Airflow and get the event generated.
My feeling is that while this is needed - it should be externalised from
airlfow webserver. The authorization has to be set up anyway additionally -
unlike in "poll" case - we cannot use Connections for authorizing (because
it's not Airlfow that authorizes in an external system - it's the other way
round). So we have to anyhow setup "something extra" in Airflow to
authorize the external system. Which could be what we have now - user that
allows us to trigger the event. Which means that our REST API could
potentially be used the same way it is now, but we will need "something"
(library, lambda function etc.) that users could easily setup in the
external system to map whatever trigger they generate natively (say S3 file
created) to Airflow REST API.
As I see it - this is quite often used (and very practical, that you deploy
a cloud function or lambda that subscribes on the event received when
S3/GCS is created. So it would be on the user to deploy such a lambda - but
we **could** provide a library of those: say s3 lambda, gcp cloud function
in respective providers - with documentation how to set them up, and how to
configure authorization and we would be generally "done". I am just not
sure if we need a new entity in Airflow for that (Event receiver). It feels
like it asks Airflow to take more responsibility, when we all think on what
to "remove" from Airflow rather than "add" to it - especially when it comes
to external integrations. It feels to me that Airflow should make it easy
to be triggered by such an external system and make it easy to "map" to the
way we expect to get events triggered, but this should be done outside of
Airflow. If the users can easily find in our docs when they search "what do
I do to externally trigger Airflow on S3 change": either a) configure
polling in airflow using s3 Connection, or b) "create a user + deploy this
lambda with those parameters" - that is "easy enough" and very practical
as well.
But maybe I am not seeing the whole picture and the real problem it's
solving - so take it as a "first review pass" and "guts feeling".
J.
On Thu, Jul 25, 2024 at 10:55 PM Beck, Vincent <vi...@amazon.com.invalid>
wrote:
> Hello everyone,
>
> I created a draft AIP regarding "External event driven scheduling in
> Airflow". This proposal is about adding capability in Airflow to schedule
> DAGs based on external events. Here are some examples of such external
> events:
> - A user signs up to one of the user pool defined in my cloud provider
> - One of the databases used in my company has been updated
> - A job in my cloud provider has been executed successfully
>
> The intent of this AIP is to leverage datasets (which will be soon assets)
> and update them based on external events. I would like to propose this AIP
> for discussion and more importantly, hear some feedbacks from you :)
>
>
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow
>
> Vincent
>