You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Dheeraj Turaga <tu...@apache.org> on 2026/10/15 04:17:22 UTC
Subject: [DISCUSS] Support Inbound webhooks for Airflow
Hi all,
I want to gauge the interest in supporting inbound webhooks for Airflow.
Why?
Webhooks are how most services announce events: source control and CI
(GitHub, GitLab, Bitbucket), data tools (dbt Cloud, Fivetran, Airbyte),
ML platforms, payment and commerce systems, incident tools (PagerDuty,
Datadog) and issue trackers (Jira, Linear). Those events are often what
a Dag is waiting for, but Airflow can't receive them: senders can't
attach an Airflow API token, they sign deliveries with a shared secret,
and each has its own payload format. So we poll with sensors, which hold
capacity and react late, or run a relay service in front of Airflow.
AssetWatcher covers pull-based queues; there's no push equivalent.
Webhooks would let a Dag start the moment a dbt Cloud job finishes, a
service is deployed, a PagerDuty incident opens or a vendor's export is
ready.
The idea:
Turn incoming webhooks into asset events that trigger your Airflow Dags!
Each verified delivery is recorded as an asset event named after what
happened, and Dags subscribe by scheduling on that name. When a pull
request in my-org/web gets the deploy-preview label, this Dag runs, with
the payload in its triggering_asset_events:
@dag(schedule=Asset("github/my-org/web/pull_request/labeled/deploy-preview"))
Broader names work too: github/my-org/*/push/main fires on a push to
main in any repository of the org. Senders don't need to know Dag IDs,
one delivery can start several Dags, and the scheduler creates the runs
as for any asset event.
Try it out:
I've built this as a provider for Airflow 3.1.1+, with GitHub, GitLab
and Jira built in, plus a generic option for any service that signs its
body with HMAC-SHA256 or sends a secret header:
https://github.com/dheerajturaga/airflow-provider-inbound-webhooks
https://pypi.org/project/airflow-provider-inbound-webhooks/
The README has a quick start that needs no third-party account, and
docs/design.md covers what could move to core.
If there's interest, I'm happy to draft an AIP. Thoughts welcome!
Thanks,
Dheeraj
[DISCUSS] Support Inbound webhooks for Airflow
Posted by Dheeraj Turaga <tu...@apache.org>.
Fixing the subject line, sorry for the noise
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Subject: [DISCUSS] Support Inbound webhooks for Airflow
Posted by "constance.astronomer.io via dev" <de...@airflow.apache.org>.
I really like the idea, but agree with Kaxil on making it more broad.
> One more thing on assets: a run picks up every event queued since the
previous run, so ten PRs labeled within a minute could end up as one run
with ten events in triggering_asset_events. I don't think people would
expect that from a webhook.
This will no longer be true as of 3.4! Unless off you opt back in to asset event batching.
Constance
> On Oct 15, 2026, at 6:59 PM, Kaxil Naik <ka...@gmail.com> wrote:
>
> One more thing on assets: a run picks up every event queued since the
> previous run, so ten PRs labeled within a minute could end up as one run
> with ten events in triggering_asset_events. I don't think people would
> expect that from a webhook.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Subject: [DISCUSS] Support Inbound webhooks for Airflow
Posted by Jarek Potiuk <ja...@potiuk.com>.
+1 to what Kaxil wrote..
On Thu, Oct 15, 2026 at 11:59 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> Hi Dheeraj,
>
> +1 on the need but I'd look at this more broadly than assets. Most of the
> webhooks I'd want Airflow to receive aren't just about data changing:
>
> - A GitHub PR gets the "deploy-preview" label: run the preview Dag for that
> PR, and cancel the run if the PR is closed before it finishes.
> - A task kicks off a dbt Cloud or Databricks job (or a day-long LLM batch)
> and defers. Right now the trigger polls the job API until it's done. If the
> service could call back instead, the task would just resume.
> - Approving an HITL task from a Slack button.
> - A Fivetran sync finished, so record an asset event. Your provider handles
> this one well.
>
> Airflow can already do every one of these through the REST API. They don't
> work today for the reason you called out: the sender can't hold a token. So
> I'd like the core piece to be verified inbound events that can drive any of
> these (asset event, new Dag run, resuming a deferred task, HITL response),
> rather than asset events only.
>
> Resuming a task or cancelling a run also means the event has to point at
> one specific run, and routing by asset name can't do that. Step Functions
> solves it by giving each waiting task its own callback token. Something
> similar here would also mean a leaked URL only affects one task, instead of
> everything behind a shared secret.
>
> One more thing on assets: a run picks up every event queued since the
> previous run, so ten PRs labeled within a minute could end up as one run
> with ten events in triggering_asset_events. I don't think people would
> expect that from a webhook.
>
> Thanks,
> Kaxil
>
> On Thu, 15 Oct 2026 at 05:17, Dheeraj Turaga <tu...@apache.org> wrote:
>
> > Hi all,
> >
> > I want to gauge the interest in supporting inbound webhooks for Airflow.
> >
> > Why?
> >
> > Webhooks are how most services announce events: source control and CI
> > (GitHub, GitLab, Bitbucket), data tools (dbt Cloud, Fivetran, Airbyte),
> > ML platforms, payment and commerce systems, incident tools (PagerDuty,
> > Datadog) and issue trackers (Jira, Linear). Those events are often what
> > a Dag is waiting for, but Airflow can't receive them: senders can't
> > attach an Airflow API token, they sign deliveries with a shared secret,
> > and each has its own payload format. So we poll with sensors, which hold
> > capacity and react late, or run a relay service in front of Airflow.
> > AssetWatcher covers pull-based queues; there's no push equivalent.
> >
> > Webhooks would let a Dag start the moment a dbt Cloud job finishes, a
> > service is deployed, a PagerDuty incident opens or a vendor's export is
> > ready.
> >
> > The idea:
> >
> > Turn incoming webhooks into asset events that trigger your Airflow Dags!
> >
> > Each verified delivery is recorded as an asset event named after what
> > happened, and Dags subscribe by scheduling on that name. When a pull
> > request in my-org/web gets the deploy-preview label, this Dag runs, with
> > the payload in its triggering_asset_events:
> >
> >
> >
> > @dag(schedule=Asset("github/my-org/web/pull_request/labeled/deploy-preview"))
> >
> > Broader names work too: github/my-org/*/push/main fires on a push to
> > main in any repository of the org. Senders don't need to know Dag IDs,
> > one delivery can start several Dags, and the scheduler creates the runs
> > as for any asset event.
> >
> > Try it out:
> >
> > I've built this as a provider for Airflow 3.1.1+, with GitHub, GitLab
> > and Jira built in, plus a generic option for any service that signs its
> > body with HMAC-SHA256 or sends a secret header:
> >
> > https://github.com/dheerajturaga/airflow-provider-inbound-webhooks
> > https://pypi.org/project/airflow-provider-inbound-webhooks/
> >
> > The README has a quick start that needs no third-party account, and
> > docs/design.md covers what could move to core.
> >
> > If there's interest, I'm happy to draft an AIP. Thoughts welcome!
> >
> > Thanks,
> > Dheeraj
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Subject: [DISCUSS] Support Inbound webhooks for Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
Hi Dheeraj,
+1 on the need but I'd look at this more broadly than assets. Most of the
webhooks I'd want Airflow to receive aren't just about data changing:
- A GitHub PR gets the "deploy-preview" label: run the preview Dag for that
PR, and cancel the run if the PR is closed before it finishes.
- A task kicks off a dbt Cloud or Databricks job (or a day-long LLM batch)
and defers. Right now the trigger polls the job API until it's done. If the
service could call back instead, the task would just resume.
- Approving an HITL task from a Slack button.
- A Fivetran sync finished, so record an asset event. Your provider handles
this one well.
Airflow can already do every one of these through the REST API. They don't
work today for the reason you called out: the sender can't hold a token. So
I'd like the core piece to be verified inbound events that can drive any of
these (asset event, new Dag run, resuming a deferred task, HITL response),
rather than asset events only.
Resuming a task or cancelling a run also means the event has to point at
one specific run, and routing by asset name can't do that. Step Functions
solves it by giving each waiting task its own callback token. Something
similar here would also mean a leaked URL only affects one task, instead of
everything behind a shared secret.
One more thing on assets: a run picks up every event queued since the
previous run, so ten PRs labeled within a minute could end up as one run
with ten events in triggering_asset_events. I don't think people would
expect that from a webhook.
Thanks,
Kaxil
On Thu, 15 Oct 2026 at 05:17, Dheeraj Turaga <tu...@apache.org> wrote:
> Hi all,
>
> I want to gauge the interest in supporting inbound webhooks for Airflow.
>
> Why?
>
> Webhooks are how most services announce events: source control and CI
> (GitHub, GitLab, Bitbucket), data tools (dbt Cloud, Fivetran, Airbyte),
> ML platforms, payment and commerce systems, incident tools (PagerDuty,
> Datadog) and issue trackers (Jira, Linear). Those events are often what
> a Dag is waiting for, but Airflow can't receive them: senders can't
> attach an Airflow API token, they sign deliveries with a shared secret,
> and each has its own payload format. So we poll with sensors, which hold
> capacity and react late, or run a relay service in front of Airflow.
> AssetWatcher covers pull-based queues; there's no push equivalent.
>
> Webhooks would let a Dag start the moment a dbt Cloud job finishes, a
> service is deployed, a PagerDuty incident opens or a vendor's export is
> ready.
>
> The idea:
>
> Turn incoming webhooks into asset events that trigger your Airflow Dags!
>
> Each verified delivery is recorded as an asset event named after what
> happened, and Dags subscribe by scheduling on that name. When a pull
> request in my-org/web gets the deploy-preview label, this Dag runs, with
> the payload in its triggering_asset_events:
>
>
>
> @dag(schedule=Asset("github/my-org/web/pull_request/labeled/deploy-preview"))
>
> Broader names work too: github/my-org/*/push/main fires on a push to
> main in any repository of the org. Senders don't need to know Dag IDs,
> one delivery can start several Dags, and the scheduler creates the runs
> as for any asset event.
>
> Try it out:
>
> I've built this as a provider for Airflow 3.1.1+, with GitHub, GitLab
> and Jira built in, plus a generic option for any service that signs its
> body with HMAC-SHA256 or sends a secret header:
>
> https://github.com/dheerajturaga/airflow-provider-inbound-webhooks
> https://pypi.org/project/airflow-provider-inbound-webhooks/
>
> The README has a quick start that needs no third-party account, and
> docs/design.md covers what could move to core.
>
> If there's interest, I'm happy to draft an AIP. Thoughts welcome!
>
> Thanks,
> Dheeraj
>