You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Niko Oliveira <on...@apache.org> on 2026/09/10 06:03:21 UTC
[DISCUSS] New Community Provider: DuckDB
Hi folks,
I'd like to propose adding a new community provider for DuckDB.
Integration overview:
---------------------
DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
providers there is no server to connect to: queries run inside the task
process, which makes it a cheap way to run an analytical transform that
doesn't justify standing up a cluster. It reads and writes Parquet, CSV and
JSON directly from object storage, so it fits naturally between an extract
and a load step in a
Dag.
It also fits Airflow's existing shape well: The proposed DuckDB provider
speaks DbAPI, so it plugs into the common-sql ecosystem rather than needing
new machinery, and it fits in the data-processing/ETL space. DuckDB Labs
was acquired by Amazon but DuckDB remains MIT licensed and controlled by
the DuckDB Foundation. It is usable with no account, no signup and no
hosted service.
The provider exposes a DuckDBHook extending DbApiHook, so it works with
SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a
DuckDBExecuteQueryOperator and a duckdb connection type. The connection is
optional: with none configured the hook opens an in-memory database.
Provider specific integrations would live in their relevant provider
package (e.g. for AWS, an AwsDuckDBHook in apache-airflow-providers-amazon
for credential management).
System tests:
-------------
DuckDB runs in-process with no live external service, so system tests are
not
required for this provider; the unit tests exercise a real in-memory DuckDB.
An AWS-specific system test could be added in the Amazon provider package,
if folks think that's required.
Proposed stewards:
------------------
Myself and the AWS team plan to sponsor and steward this package, you know
us and our track record well :)
- Niko Oliveira (@o-nikolas)
- Vincent Beck (@vincbeck)
Working implementation
----------------------
https://github.com/apache/airflow/pull/72845
Incubation commitment
---------------------
We commit to:
- Maintaining the provider and responding to issues within a reasonable time
- Meeting the incubation health metrics within 6 months
- Participating in quarterly governance updates
Cheers,
Niko
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Niko Oliveira <on...@apache.org>.
Hey folks!
Just noting that the 7 day discussion period is over and I have merged the
PR for the DuckDB provider :)
Thanks for the engagement!
Cheers,
Niko
On Sat, Sep 12, 2026 at 9:07 AM Aaron Chen <na...@gmail.com> wrote:
> +1
>
> On Sat, Sep 12, 2026 at 11:46 AM Wei Lee <we...@apache.org> wrote:
>
> > +1
> >
> > On 2026/09/10 23:54:59 Vikram Koka via dev wrote:
> > > +1
> > >
> > > On Thu, Sep 10, 2026 at 3:43 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > +1
> > > >
> > > > On Thu, 10 Sept 2026 at 18:04, Shubham Raj <sh...@apache.org>
> > wrote:
> > > >
> > > > > Sounds good, +1
> > > > >
> > > > > On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <
> smesiah971@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Arguably, long overdue.
> > > > > >
> > > > > > +1 from me. I will review the PR and offer my thoughts if that
> is
> > ok
> > > > > with
> > > > > > you.
> > > > > >
> > > > > > Thanks,
> > > > > > Sameer.
> > > > > >
> > > > > > On Thu, 10 Sep 2026 at 16:38, Amogh Desai <amoghdesai@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Sounds great, let's bring it in!
> > > > > > >
> > > > > > > Thanks & Regards,
> > > > > > > Amogh Desai
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <
> > onikolas@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Julian, much appreciated!
> > > > > > > >
> > > > > > > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > > > > > > > dev@airflow.apache.org> wrote:
> > > > > > > >
> > > > > > > > > I wrote our original DuckDB provider at Astronomer, would
> be
> > more
> > > > > > than
> > > > > > > > > happy to deprecate in favor of yours! Definitely would love
> > to
> > > > see
> > > > > > > > > something more first-class and better supported.
> > > > > > > > >
> > > > > > > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <
> > > > onikolas@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Glad to see the traction, thanks for weighing in
> everyone!
> > > > > > > > > >
> > > > > > > > > >> Is this based on the Astronomer one?
> > > > > > > > > > https://github.com/astronomer/airflow-provider-duckdb
> > > > > > > > > >
> > > > > > > > > > No code was based on that provider. It hasn't been
> updated
> > in 3
> > > > > > years
> > > > > > > > and
> > > > > > > > > > it focused on a minimal set of DuckDB features. There
> > wasn't
> > > > much
> > > > > > to
> > > > > > > > > reuse
> > > > > > > > > > there.
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <
> > jarek@potiuk.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> +10
> > > > > > > > > >>
> > > > > > > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <
> gid.cto@gmail.com
> > >
> > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Sounds good!
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <
> > > > onikolas@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hi folks,
> > > > > > > > > >>>>
> > > > > > > > > >>>> I'd like to propose adding a new community provider
> for
> > > > > DuckDB.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Integration overview:
> > > > > > > > > >>>> ---------------------
> > > > > > > > > >>>>
> > > > > > > > > >>>> DuckDB is an in-process analytical (OLAP) database.
> > Unlike
> > > > > most
> > > > > > > SQL
> > > > > > > > > >>>> providers there is no server to connect to: queries
> run
> > > > inside
> > > > > > the
> > > > > > > > > task
> > > > > > > > > >>>> process, which makes it a cheap way to run an
> analytical
> > > > > > transform
> > > > > > > > > that
> > > > > > > > > >>>> doesn't justify standing up a cluster. It reads and
> > writes
> > > > > > > Parquet,
> > > > > > > > > CSV
> > > > > > > > > >>> and
> > > > > > > > > >>>> JSON directly from object storage, so it fits
> naturally
> > > > > between
> > > > > > an
> > > > > > > > > >>> extract
> > > > > > > > > >>>> and a load step in a
> > > > > > > > > >>>> Dag.
> > > > > > > > > >>>>
> > > > > > > > > >>>> It also fits Airflow's existing shape well: The
> proposed
> > > > > DuckDB
> > > > > > > > > >> provider
> > > > > > > > > >>>> speaks DbAPI, so it plugs into the common-sql
> ecosystem
> > > > rather
> > > > > > > than
> > > > > > > > > >>> needing
> > > > > > > > > >>>> new machinery, and it fits in the data-processing/ETL
> > space.
> > > > > > > DuckDB
> > > > > > > > > >> Labs
> > > > > > > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed
> > and
> > > > > > > > controlled
> > > > > > > > > >> by
> > > > > > > > > >>>> the DuckDB Foundation. It is usable with no account,
> no
> > > > signup
> > > > > > and
> > > > > > > > no
> > > > > > > > > >>>> hosted service.
> > > > > > > > > >>>>
> > > > > > > > > >>>> The provider exposes a DuckDBHook extending DbApiHook,
> > so it
> > > > > > works
> > > > > > > > > with
> > > > > > > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql
> > > > > > ecosystem,
> > > > > > > > plus
> > > > > > > > > >> a
> > > > > > > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection
> > type. The
> > > > > > > > > connection
> > > > > > > > > >>> is
> > > > > > > > > >>>> optional: with none configured the hook opens an
> > in-memory
> > > > > > > database.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Provider specific integrations would live in their
> > relevant
> > > > > > > provider
> > > > > > > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > > > > > > > >>> apache-airflow-providers-amazon
> > > > > > > > > >>>> for credential management).
> > > > > > > > > >>>>
> > > > > > > > > >>>> System tests:
> > > > > > > > > >>>> -------------
> > > > > > > > > >>>>
> > > > > > > > > >>>> DuckDB runs in-process with no live external service,
> so
> > > > > system
> > > > > > > > tests
> > > > > > > > > >> are
> > > > > > > > > >>>> not
> > > > > > > > > >>>> required for this provider; the unit tests exercise a
> > real
> > > > > > > in-memory
> > > > > > > > > >>>> DuckDB.
> > > > > > > > > >>>> An AWS-specific system test could be added in the
> Amazon
> > > > > > provider
> > > > > > > > > >>> package,
> > > > > > > > > >>>> if folks think that's required.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Proposed stewards:
> > > > > > > > > >>>> ------------------
> > > > > > > > > >>>>
> > > > > > > > > >>>> Myself and the AWS team plan to sponsor and steward
> this
> > > > > > package,
> > > > > > > > you
> > > > > > > > > >>> know
> > > > > > > > > >>>> us and our track record well :)
> > > > > > > > > >>>>
> > > > > > > > > >>>> - Niko Oliveira (@o-nikolas)
> > > > > > > > > >>>> - Vincent Beck (@vincbeck)
> > > > > > > > > >>>>
> > > > > > > > > >>>> Working implementation
> > > > > > > > > >>>> ----------------------
> > > > > > > > > >>>>
> > > > > > > > > >>>> https://github.com/apache/airflow/pull/72845
> > > > > > > > > >>>>
> > > > > > > > > >>>> Incubation commitment
> > > > > > > > > >>>> ---------------------
> > > > > > > > > >>>>
> > > > > > > > > >>>> We commit to:
> > > > > > > > > >>>>
> > > > > > > > > >>>> - Maintaining the provider and responding to issues
> > within a
> > > > > > > > > reasonable
> > > > > > > > > >>>> time
> > > > > > > > > >>>> - Meeting the incubation health metrics within 6
> months
> > > > > > > > > >>>> - Participating in quarterly governance updates
> > > > > > > > > >>>>
> > > > > > > > > >>>> Cheers,
> > > > > > > > > >>>> Niko
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Aaron Chen <na...@gmail.com>.
+1
On Sat, Sep 12, 2026 at 11:46 AM Wei Lee <we...@apache.org> wrote:
> +1
>
> On 2026/09/10 23:54:59 Vikram Koka via dev wrote:
> > +1
> >
> > On Thu, Sep 10, 2026 at 3:43 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Thu, 10 Sept 2026 at 18:04, Shubham Raj <sh...@apache.org>
> wrote:
> > >
> > > > Sounds good, +1
> > > >
> > > > On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <sm...@gmail.com>
> > > > wrote:
> > > >
> > > > > Arguably, long overdue.
> > > > >
> > > > > +1 from me. I will review the PR and offer my thoughts if that is
> ok
> > > > with
> > > > > you.
> > > > >
> > > > > Thanks,
> > > > > Sameer.
> > > > >
> > > > > On Thu, 10 Sep 2026 at 16:38, Amogh Desai <am...@apache.org>
> > > wrote:
> > > > >
> > > > > > Sounds great, let's bring it in!
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Amogh Desai
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <
> onikolas@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Julian, much appreciated!
> > > > > > >
> > > > > > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > > > > > > dev@airflow.apache.org> wrote:
> > > > > > >
> > > > > > > > I wrote our original DuckDB provider at Astronomer, would be
> more
> > > > > than
> > > > > > > > happy to deprecate in favor of yours! Definitely would love
> to
> > > see
> > > > > > > > something more first-class and better supported.
> > > > > > > >
> > > > > > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <
> > > onikolas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Glad to see the traction, thanks for weighing in everyone!
> > > > > > > > >
> > > > > > > > >> Is this based on the Astronomer one?
> > > > > > > > > https://github.com/astronomer/airflow-provider-duckdb
> > > > > > > > >
> > > > > > > > > No code was based on that provider. It hasn't been updated
> in 3
> > > > > years
> > > > > > > and
> > > > > > > > > it focused on a minimal set of DuckDB features. There
> wasn't
> > > much
> > > > > to
> > > > > > > > reuse
> > > > > > > > > there.
> > > > > > > > >
> > > > > > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <
> jarek@potiuk.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> +10
> > > > > > > > >>
> > > > > > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gid.cto@gmail.com
> >
> > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Sounds good!
> > > > > > > > >>>
> > > > > > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <
> > > onikolas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hi folks,
> > > > > > > > >>>>
> > > > > > > > >>>> I'd like to propose adding a new community provider for
> > > > DuckDB.
> > > > > > > > >>>>
> > > > > > > > >>>> Integration overview:
> > > > > > > > >>>> ---------------------
> > > > > > > > >>>>
> > > > > > > > >>>> DuckDB is an in-process analytical (OLAP) database.
> Unlike
> > > > most
> > > > > > SQL
> > > > > > > > >>>> providers there is no server to connect to: queries run
> > > inside
> > > > > the
> > > > > > > > task
> > > > > > > > >>>> process, which makes it a cheap way to run an analytical
> > > > > transform
> > > > > > > > that
> > > > > > > > >>>> doesn't justify standing up a cluster. It reads and
> writes
> > > > > > Parquet,
> > > > > > > > CSV
> > > > > > > > >>> and
> > > > > > > > >>>> JSON directly from object storage, so it fits naturally
> > > > between
> > > > > an
> > > > > > > > >>> extract
> > > > > > > > >>>> and a load step in a
> > > > > > > > >>>> Dag.
> > > > > > > > >>>>
> > > > > > > > >>>> It also fits Airflow's existing shape well: The proposed
> > > > DuckDB
> > > > > > > > >> provider
> > > > > > > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem
> > > rather
> > > > > > than
> > > > > > > > >>> needing
> > > > > > > > >>>> new machinery, and it fits in the data-processing/ETL
> space.
> > > > > > DuckDB
> > > > > > > > >> Labs
> > > > > > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed
> and
> > > > > > > controlled
> > > > > > > > >> by
> > > > > > > > >>>> the DuckDB Foundation. It is usable with no account, no
> > > signup
> > > > > and
> > > > > > > no
> > > > > > > > >>>> hosted service.
> > > > > > > > >>>>
> > > > > > > > >>>> The provider exposes a DuckDBHook extending DbApiHook,
> so it
> > > > > works
> > > > > > > > with
> > > > > > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql
> > > > > ecosystem,
> > > > > > > plus
> > > > > > > > >> a
> > > > > > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection
> type. The
> > > > > > > > connection
> > > > > > > > >>> is
> > > > > > > > >>>> optional: with none configured the hook opens an
> in-memory
> > > > > > database.
> > > > > > > > >>>>
> > > > > > > > >>>> Provider specific integrations would live in their
> relevant
> > > > > > provider
> > > > > > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > > > > > > >>> apache-airflow-providers-amazon
> > > > > > > > >>>> for credential management).
> > > > > > > > >>>>
> > > > > > > > >>>> System tests:
> > > > > > > > >>>> -------------
> > > > > > > > >>>>
> > > > > > > > >>>> DuckDB runs in-process with no live external service, so
> > > > system
> > > > > > > tests
> > > > > > > > >> are
> > > > > > > > >>>> not
> > > > > > > > >>>> required for this provider; the unit tests exercise a
> real
> > > > > > in-memory
> > > > > > > > >>>> DuckDB.
> > > > > > > > >>>> An AWS-specific system test could be added in the Amazon
> > > > > provider
> > > > > > > > >>> package,
> > > > > > > > >>>> if folks think that's required.
> > > > > > > > >>>>
> > > > > > > > >>>> Proposed stewards:
> > > > > > > > >>>> ------------------
> > > > > > > > >>>>
> > > > > > > > >>>> Myself and the AWS team plan to sponsor and steward this
> > > > > package,
> > > > > > > you
> > > > > > > > >>> know
> > > > > > > > >>>> us and our track record well :)
> > > > > > > > >>>>
> > > > > > > > >>>> - Niko Oliveira (@o-nikolas)
> > > > > > > > >>>> - Vincent Beck (@vincbeck)
> > > > > > > > >>>>
> > > > > > > > >>>> Working implementation
> > > > > > > > >>>> ----------------------
> > > > > > > > >>>>
> > > > > > > > >>>> https://github.com/apache/airflow/pull/72845
> > > > > > > > >>>>
> > > > > > > > >>>> Incubation commitment
> > > > > > > > >>>> ---------------------
> > > > > > > > >>>>
> > > > > > > > >>>> We commit to:
> > > > > > > > >>>>
> > > > > > > > >>>> - Maintaining the provider and responding to issues
> within a
> > > > > > > > reasonable
> > > > > > > > >>>> time
> > > > > > > > >>>> - Meeting the incubation health metrics within 6 months
> > > > > > > > >>>> - Participating in quarterly governance updates
> > > > > > > > >>>>
> > > > > > > > >>>> Cheers,
> > > > > > > > >>>> Niko
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Wei Lee <we...@apache.org>.
+1
On 2026/09/10 23:54:59 Vikram Koka via dev wrote:
> +1
>
> On Thu, Sep 10, 2026 at 3:43 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > +1
> >
> > On Thu, 10 Sept 2026 at 18:04, Shubham Raj <sh...@apache.org> wrote:
> >
> > > Sounds good, +1
> > >
> > > On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <sm...@gmail.com>
> > > wrote:
> > >
> > > > Arguably, long overdue.
> > > >
> > > > +1 from me. I will review the PR and offer my thoughts if that is ok
> > > with
> > > > you.
> > > >
> > > > Thanks,
> > > > Sameer.
> > > >
> > > > On Thu, 10 Sep 2026 at 16:38, Amogh Desai <am...@apache.org>
> > wrote:
> > > >
> > > > > Sounds great, let's bring it in!
> > > > >
> > > > > Thanks & Regards,
> > > > > Amogh Desai
> > > > >
> > > > >
> > > > > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <on...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Thanks Julian, much appreciated!
> > > > > >
> > > > > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > > > > > dev@airflow.apache.org> wrote:
> > > > > >
> > > > > > > I wrote our original DuckDB provider at Astronomer, would be more
> > > > than
> > > > > > > happy to deprecate in favor of yours! Definitely would love to
> > see
> > > > > > > something more first-class and better supported.
> > > > > > >
> > > > > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <
> > onikolas@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Glad to see the traction, thanks for weighing in everyone!
> > > > > > > >
> > > > > > > >> Is this based on the Astronomer one?
> > > > > > > > https://github.com/astronomer/airflow-provider-duckdb
> > > > > > > >
> > > > > > > > No code was based on that provider. It hasn't been updated in 3
> > > > years
> > > > > > and
> > > > > > > > it focused on a minimal set of DuckDB features. There wasn't
> > much
> > > > to
> > > > > > > reuse
> > > > > > > > there.
> > > > > > > >
> > > > > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <jarek@potiuk.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > >> +10
> > > > > > > >>
> > > > > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com>
> > > wrote:
> > > > > > > >>
> > > > > > > >>> Sounds good!
> > > > > > > >>>
> > > > > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <
> > onikolas@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >>>
> > > > > > > >>>> Hi folks,
> > > > > > > >>>>
> > > > > > > >>>> I'd like to propose adding a new community provider for
> > > DuckDB.
> > > > > > > >>>>
> > > > > > > >>>> Integration overview:
> > > > > > > >>>> ---------------------
> > > > > > > >>>>
> > > > > > > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike
> > > most
> > > > > SQL
> > > > > > > >>>> providers there is no server to connect to: queries run
> > inside
> > > > the
> > > > > > > task
> > > > > > > >>>> process, which makes it a cheap way to run an analytical
> > > > transform
> > > > > > > that
> > > > > > > >>>> doesn't justify standing up a cluster. It reads and writes
> > > > > Parquet,
> > > > > > > CSV
> > > > > > > >>> and
> > > > > > > >>>> JSON directly from object storage, so it fits naturally
> > > between
> > > > an
> > > > > > > >>> extract
> > > > > > > >>>> and a load step in a
> > > > > > > >>>> Dag.
> > > > > > > >>>>
> > > > > > > >>>> It also fits Airflow's existing shape well: The proposed
> > > DuckDB
> > > > > > > >> provider
> > > > > > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem
> > rather
> > > > > than
> > > > > > > >>> needing
> > > > > > > >>>> new machinery, and it fits in the data-processing/ETL space.
> > > > > DuckDB
> > > > > > > >> Labs
> > > > > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed and
> > > > > > controlled
> > > > > > > >> by
> > > > > > > >>>> the DuckDB Foundation. It is usable with no account, no
> > signup
> > > > and
> > > > > > no
> > > > > > > >>>> hosted service.
> > > > > > > >>>>
> > > > > > > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it
> > > > works
> > > > > > > with
> > > > > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql
> > > > ecosystem,
> > > > > > plus
> > > > > > > >> a
> > > > > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> > > > > > > connection
> > > > > > > >>> is
> > > > > > > >>>> optional: with none configured the hook opens an in-memory
> > > > > database.
> > > > > > > >>>>
> > > > > > > >>>> Provider specific integrations would live in their relevant
> > > > > provider
> > > > > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > > > > > >>> apache-airflow-providers-amazon
> > > > > > > >>>> for credential management).
> > > > > > > >>>>
> > > > > > > >>>> System tests:
> > > > > > > >>>> -------------
> > > > > > > >>>>
> > > > > > > >>>> DuckDB runs in-process with no live external service, so
> > > system
> > > > > > tests
> > > > > > > >> are
> > > > > > > >>>> not
> > > > > > > >>>> required for this provider; the unit tests exercise a real
> > > > > in-memory
> > > > > > > >>>> DuckDB.
> > > > > > > >>>> An AWS-specific system test could be added in the Amazon
> > > > provider
> > > > > > > >>> package,
> > > > > > > >>>> if folks think that's required.
> > > > > > > >>>>
> > > > > > > >>>> Proposed stewards:
> > > > > > > >>>> ------------------
> > > > > > > >>>>
> > > > > > > >>>> Myself and the AWS team plan to sponsor and steward this
> > > > package,
> > > > > > you
> > > > > > > >>> know
> > > > > > > >>>> us and our track record well :)
> > > > > > > >>>>
> > > > > > > >>>> - Niko Oliveira (@o-nikolas)
> > > > > > > >>>> - Vincent Beck (@vincbeck)
> > > > > > > >>>>
> > > > > > > >>>> Working implementation
> > > > > > > >>>> ----------------------
> > > > > > > >>>>
> > > > > > > >>>> https://github.com/apache/airflow/pull/72845
> > > > > > > >>>>
> > > > > > > >>>> Incubation commitment
> > > > > > > >>>> ---------------------
> > > > > > > >>>>
> > > > > > > >>>> We commit to:
> > > > > > > >>>>
> > > > > > > >>>> - Maintaining the provider and responding to issues within a
> > > > > > > reasonable
> > > > > > > >>>> time
> > > > > > > >>>> - Meeting the incubation health metrics within 6 months
> > > > > > > >>>> - Participating in quarterly governance updates
> > > > > > > >>>>
> > > > > > > >>>> Cheers,
> > > > > > > >>>> Niko
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Vikram Koka via dev <de...@airflow.apache.org>.
+1
On Thu, Sep 10, 2026 at 3:43 PM Kaxil Naik <ka...@gmail.com> wrote:
> +1
>
> On Thu, 10 Sept 2026 at 18:04, Shubham Raj <sh...@apache.org> wrote:
>
> > Sounds good, +1
> >
> > On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <sm...@gmail.com>
> > wrote:
> >
> > > Arguably, long overdue.
> > >
> > > +1 from me. I will review the PR and offer my thoughts if that is ok
> > with
> > > you.
> > >
> > > Thanks,
> > > Sameer.
> > >
> > > On Thu, 10 Sep 2026 at 16:38, Amogh Desai <am...@apache.org>
> wrote:
> > >
> > > > Sounds great, let's bring it in!
> > > >
> > > > Thanks & Regards,
> > > > Amogh Desai
> > > >
> > > >
> > > > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <on...@apache.org>
> > > wrote:
> > > >
> > > > > Thanks Julian, much appreciated!
> > > > >
> > > > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > > > > dev@airflow.apache.org> wrote:
> > > > >
> > > > > > I wrote our original DuckDB provider at Astronomer, would be more
> > > than
> > > > > > happy to deprecate in favor of yours! Definitely would love to
> see
> > > > > > something more first-class and better supported.
> > > > > >
> > > > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <
> onikolas@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > Glad to see the traction, thanks for weighing in everyone!
> > > > > > >
> > > > > > >> Is this based on the Astronomer one?
> > > > > > > https://github.com/astronomer/airflow-provider-duckdb
> > > > > > >
> > > > > > > No code was based on that provider. It hasn't been updated in 3
> > > years
> > > > > and
> > > > > > > it focused on a minimal set of DuckDB features. There wasn't
> much
> > > to
> > > > > > reuse
> > > > > > > there.
> > > > > > >
> > > > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <jarek@potiuk.com
> >
> > > > wrote:
> > > > > > >
> > > > > > >> +10
> > > > > > >>
> > > > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com>
> > wrote:
> > > > > > >>
> > > > > > >>> Sounds good!
> > > > > > >>>
> > > > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <
> onikolas@apache.org
> > >
> > > > > wrote:
> > > > > > >>>
> > > > > > >>>> Hi folks,
> > > > > > >>>>
> > > > > > >>>> I'd like to propose adding a new community provider for
> > DuckDB.
> > > > > > >>>>
> > > > > > >>>> Integration overview:
> > > > > > >>>> ---------------------
> > > > > > >>>>
> > > > > > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike
> > most
> > > > SQL
> > > > > > >>>> providers there is no server to connect to: queries run
> inside
> > > the
> > > > > > task
> > > > > > >>>> process, which makes it a cheap way to run an analytical
> > > transform
> > > > > > that
> > > > > > >>>> doesn't justify standing up a cluster. It reads and writes
> > > > Parquet,
> > > > > > CSV
> > > > > > >>> and
> > > > > > >>>> JSON directly from object storage, so it fits naturally
> > between
> > > an
> > > > > > >>> extract
> > > > > > >>>> and a load step in a
> > > > > > >>>> Dag.
> > > > > > >>>>
> > > > > > >>>> It also fits Airflow's existing shape well: The proposed
> > DuckDB
> > > > > > >> provider
> > > > > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem
> rather
> > > > than
> > > > > > >>> needing
> > > > > > >>>> new machinery, and it fits in the data-processing/ETL space.
> > > > DuckDB
> > > > > > >> Labs
> > > > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed and
> > > > > controlled
> > > > > > >> by
> > > > > > >>>> the DuckDB Foundation. It is usable with no account, no
> signup
> > > and
> > > > > no
> > > > > > >>>> hosted service.
> > > > > > >>>>
> > > > > > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it
> > > works
> > > > > > with
> > > > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql
> > > ecosystem,
> > > > > plus
> > > > > > >> a
> > > > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> > > > > > connection
> > > > > > >>> is
> > > > > > >>>> optional: with none configured the hook opens an in-memory
> > > > database.
> > > > > > >>>>
> > > > > > >>>> Provider specific integrations would live in their relevant
> > > > provider
> > > > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > > > > >>> apache-airflow-providers-amazon
> > > > > > >>>> for credential management).
> > > > > > >>>>
> > > > > > >>>> System tests:
> > > > > > >>>> -------------
> > > > > > >>>>
> > > > > > >>>> DuckDB runs in-process with no live external service, so
> > system
> > > > > tests
> > > > > > >> are
> > > > > > >>>> not
> > > > > > >>>> required for this provider; the unit tests exercise a real
> > > > in-memory
> > > > > > >>>> DuckDB.
> > > > > > >>>> An AWS-specific system test could be added in the Amazon
> > > provider
> > > > > > >>> package,
> > > > > > >>>> if folks think that's required.
> > > > > > >>>>
> > > > > > >>>> Proposed stewards:
> > > > > > >>>> ------------------
> > > > > > >>>>
> > > > > > >>>> Myself and the AWS team plan to sponsor and steward this
> > > package,
> > > > > you
> > > > > > >>> know
> > > > > > >>>> us and our track record well :)
> > > > > > >>>>
> > > > > > >>>> - Niko Oliveira (@o-nikolas)
> > > > > > >>>> - Vincent Beck (@vincbeck)
> > > > > > >>>>
> > > > > > >>>> Working implementation
> > > > > > >>>> ----------------------
> > > > > > >>>>
> > > > > > >>>> https://github.com/apache/airflow/pull/72845
> > > > > > >>>>
> > > > > > >>>> Incubation commitment
> > > > > > >>>> ---------------------
> > > > > > >>>>
> > > > > > >>>> We commit to:
> > > > > > >>>>
> > > > > > >>>> - Maintaining the provider and responding to issues within a
> > > > > > reasonable
> > > > > > >>>> time
> > > > > > >>>> - Meeting the incubation health metrics within 6 months
> > > > > > >>>> - Participating in quarterly governance updates
> > > > > > >>>>
> > > > > > >>>> Cheers,
> > > > > > >>>> Niko
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Kaxil Naik <ka...@gmail.com>.
+1
On Thu, 10 Sept 2026 at 18:04, Shubham Raj <sh...@apache.org> wrote:
> Sounds good, +1
>
> On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <sm...@gmail.com>
> wrote:
>
> > Arguably, long overdue.
> >
> > +1 from me. I will review the PR and offer my thoughts if that is ok
> with
> > you.
> >
> > Thanks,
> > Sameer.
> >
> > On Thu, 10 Sep 2026 at 16:38, Amogh Desai <am...@apache.org> wrote:
> >
> > > Sounds great, let's bring it in!
> > >
> > > Thanks & Regards,
> > > Amogh Desai
> > >
> > >
> > > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <on...@apache.org>
> > wrote:
> > >
> > > > Thanks Julian, much appreciated!
> > > >
> > > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > > > dev@airflow.apache.org> wrote:
> > > >
> > > > > I wrote our original DuckDB provider at Astronomer, would be more
> > than
> > > > > happy to deprecate in favor of yours! Definitely would love to see
> > > > > something more first-class and better supported.
> > > > >
> > > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <onikolas@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > Glad to see the traction, thanks for weighing in everyone!
> > > > > >
> > > > > >> Is this based on the Astronomer one?
> > > > > > https://github.com/astronomer/airflow-provider-duckdb
> > > > > >
> > > > > > No code was based on that provider. It hasn't been updated in 3
> > years
> > > > and
> > > > > > it focused on a minimal set of DuckDB features. There wasn't much
> > to
> > > > > reuse
> > > > > > there.
> > > > > >
> > > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > > >
> > > > > >> +10
> > > > > >>
> > > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com>
> wrote:
> > > > > >>
> > > > > >>> Sounds good!
> > > > > >>>
> > > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <onikolas@apache.org
> >
> > > > wrote:
> > > > > >>>
> > > > > >>>> Hi folks,
> > > > > >>>>
> > > > > >>>> I'd like to propose adding a new community provider for
> DuckDB.
> > > > > >>>>
> > > > > >>>> Integration overview:
> > > > > >>>> ---------------------
> > > > > >>>>
> > > > > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike
> most
> > > SQL
> > > > > >>>> providers there is no server to connect to: queries run inside
> > the
> > > > > task
> > > > > >>>> process, which makes it a cheap way to run an analytical
> > transform
> > > > > that
> > > > > >>>> doesn't justify standing up a cluster. It reads and writes
> > > Parquet,
> > > > > CSV
> > > > > >>> and
> > > > > >>>> JSON directly from object storage, so it fits naturally
> between
> > an
> > > > > >>> extract
> > > > > >>>> and a load step in a
> > > > > >>>> Dag.
> > > > > >>>>
> > > > > >>>> It also fits Airflow's existing shape well: The proposed
> DuckDB
> > > > > >> provider
> > > > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather
> > > than
> > > > > >>> needing
> > > > > >>>> new machinery, and it fits in the data-processing/ETL space.
> > > DuckDB
> > > > > >> Labs
> > > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed and
> > > > controlled
> > > > > >> by
> > > > > >>>> the DuckDB Foundation. It is usable with no account, no signup
> > and
> > > > no
> > > > > >>>> hosted service.
> > > > > >>>>
> > > > > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it
> > works
> > > > > with
> > > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql
> > ecosystem,
> > > > plus
> > > > > >> a
> > > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> > > > > connection
> > > > > >>> is
> > > > > >>>> optional: with none configured the hook opens an in-memory
> > > database.
> > > > > >>>>
> > > > > >>>> Provider specific integrations would live in their relevant
> > > provider
> > > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > > > >>> apache-airflow-providers-amazon
> > > > > >>>> for credential management).
> > > > > >>>>
> > > > > >>>> System tests:
> > > > > >>>> -------------
> > > > > >>>>
> > > > > >>>> DuckDB runs in-process with no live external service, so
> system
> > > > tests
> > > > > >> are
> > > > > >>>> not
> > > > > >>>> required for this provider; the unit tests exercise a real
> > > in-memory
> > > > > >>>> DuckDB.
> > > > > >>>> An AWS-specific system test could be added in the Amazon
> > provider
> > > > > >>> package,
> > > > > >>>> if folks think that's required.
> > > > > >>>>
> > > > > >>>> Proposed stewards:
> > > > > >>>> ------------------
> > > > > >>>>
> > > > > >>>> Myself and the AWS team plan to sponsor and steward this
> > package,
> > > > you
> > > > > >>> know
> > > > > >>>> us and our track record well :)
> > > > > >>>>
> > > > > >>>> - Niko Oliveira (@o-nikolas)
> > > > > >>>> - Vincent Beck (@vincbeck)
> > > > > >>>>
> > > > > >>>> Working implementation
> > > > > >>>> ----------------------
> > > > > >>>>
> > > > > >>>> https://github.com/apache/airflow/pull/72845
> > > > > >>>>
> > > > > >>>> Incubation commitment
> > > > > >>>> ---------------------
> > > > > >>>>
> > > > > >>>> We commit to:
> > > > > >>>>
> > > > > >>>> - Maintaining the provider and responding to issues within a
> > > > > reasonable
> > > > > >>>> time
> > > > > >>>> - Meeting the incubation health metrics within 6 months
> > > > > >>>> - Participating in quarterly governance updates
> > > > > >>>>
> > > > > >>>> Cheers,
> > > > > >>>> Niko
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Shubham Raj <sh...@apache.org>.
Sounds good, +1
On Thu, Sep 10, 2026 at 9:44 AM Sameer Mesiah <sm...@gmail.com> wrote:
> Arguably, long overdue.
>
> +1 from me. I will review the PR and offer my thoughts if that is ok with
> you.
>
> Thanks,
> Sameer.
>
> On Thu, 10 Sep 2026 at 16:38, Amogh Desai <am...@apache.org> wrote:
>
> > Sounds great, let's bring it in!
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <on...@apache.org>
> wrote:
> >
> > > Thanks Julian, much appreciated!
> > >
> > > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > > dev@airflow.apache.org> wrote:
> > >
> > > > I wrote our original DuckDB provider at Astronomer, would be more
> than
> > > > happy to deprecate in favor of yours! Definitely would love to see
> > > > something more first-class and better supported.
> > > >
> > > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <on...@apache.org>
> > > wrote:
> > > > >
> > > > > Glad to see the traction, thanks for weighing in everyone!
> > > > >
> > > > >> Is this based on the Astronomer one?
> > > > > https://github.com/astronomer/airflow-provider-duckdb
> > > > >
> > > > > No code was based on that provider. It hasn't been updated in 3
> years
> > > and
> > > > > it focused on a minimal set of DuckDB features. There wasn't much
> to
> > > > reuse
> > > > > there.
> > > > >
> > > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >
> > > > >> +10
> > > > >>
> > > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
> > > > >>
> > > > >>> Sounds good!
> > > > >>>
> > > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org>
> > > wrote:
> > > > >>>
> > > > >>>> Hi folks,
> > > > >>>>
> > > > >>>> I'd like to propose adding a new community provider for DuckDB.
> > > > >>>>
> > > > >>>> Integration overview:
> > > > >>>> ---------------------
> > > > >>>>
> > > > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike most
> > SQL
> > > > >>>> providers there is no server to connect to: queries run inside
> the
> > > > task
> > > > >>>> process, which makes it a cheap way to run an analytical
> transform
> > > > that
> > > > >>>> doesn't justify standing up a cluster. It reads and writes
> > Parquet,
> > > > CSV
> > > > >>> and
> > > > >>>> JSON directly from object storage, so it fits naturally between
> an
> > > > >>> extract
> > > > >>>> and a load step in a
> > > > >>>> Dag.
> > > > >>>>
> > > > >>>> It also fits Airflow's existing shape well: The proposed DuckDB
> > > > >> provider
> > > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather
> > than
> > > > >>> needing
> > > > >>>> new machinery, and it fits in the data-processing/ETL space.
> > DuckDB
> > > > >> Labs
> > > > >>>> was acquired by Amazon but DuckDB remains MIT licensed and
> > > controlled
> > > > >> by
> > > > >>>> the DuckDB Foundation. It is usable with no account, no signup
> and
> > > no
> > > > >>>> hosted service.
> > > > >>>>
> > > > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it
> works
> > > > with
> > > > >>>> SQLExecuteQueryOperator and the rest of the common-sql
> ecosystem,
> > > plus
> > > > >> a
> > > > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> > > > connection
> > > > >>> is
> > > > >>>> optional: with none configured the hook opens an in-memory
> > database.
> > > > >>>>
> > > > >>>> Provider specific integrations would live in their relevant
> > provider
> > > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > > >>> apache-airflow-providers-amazon
> > > > >>>> for credential management).
> > > > >>>>
> > > > >>>> System tests:
> > > > >>>> -------------
> > > > >>>>
> > > > >>>> DuckDB runs in-process with no live external service, so system
> > > tests
> > > > >> are
> > > > >>>> not
> > > > >>>> required for this provider; the unit tests exercise a real
> > in-memory
> > > > >>>> DuckDB.
> > > > >>>> An AWS-specific system test could be added in the Amazon
> provider
> > > > >>> package,
> > > > >>>> if folks think that's required.
> > > > >>>>
> > > > >>>> Proposed stewards:
> > > > >>>> ------------------
> > > > >>>>
> > > > >>>> Myself and the AWS team plan to sponsor and steward this
> package,
> > > you
> > > > >>> know
> > > > >>>> us and our track record well :)
> > > > >>>>
> > > > >>>> - Niko Oliveira (@o-nikolas)
> > > > >>>> - Vincent Beck (@vincbeck)
> > > > >>>>
> > > > >>>> Working implementation
> > > > >>>> ----------------------
> > > > >>>>
> > > > >>>> https://github.com/apache/airflow/pull/72845
> > > > >>>>
> > > > >>>> Incubation commitment
> > > > >>>> ---------------------
> > > > >>>>
> > > > >>>> We commit to:
> > > > >>>>
> > > > >>>> - Maintaining the provider and responding to issues within a
> > > > reasonable
> > > > >>>> time
> > > > >>>> - Meeting the incubation health metrics within 6 months
> > > > >>>> - Participating in quarterly governance updates
> > > > >>>>
> > > > >>>> Cheers,
> > > > >>>> Niko
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Sameer Mesiah <sm...@gmail.com>.
Arguably, long overdue.
+1 from me. I will review the PR and offer my thoughts if that is ok with
you.
Thanks,
Sameer.
On Thu, 10 Sep 2026 at 16:38, Amogh Desai <am...@apache.org> wrote:
> Sounds great, let's bring it in!
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <on...@apache.org> wrote:
>
> > Thanks Julian, much appreciated!
> >
> > On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> > dev@airflow.apache.org> wrote:
> >
> > > I wrote our original DuckDB provider at Astronomer, would be more than
> > > happy to deprecate in favor of yours! Definitely would love to see
> > > something more first-class and better supported.
> > >
> > > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <on...@apache.org>
> > wrote:
> > > >
> > > > Glad to see the traction, thanks for weighing in everyone!
> > > >
> > > >> Is this based on the Astronomer one?
> > > > https://github.com/astronomer/airflow-provider-duckdb
> > > >
> > > > No code was based on that provider. It hasn't been updated in 3 years
> > and
> > > > it focused on a minimal set of DuckDB features. There wasn't much to
> > > reuse
> > > > there.
> > > >
> > > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > >> +10
> > > >>
> > > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
> > > >>
> > > >>> Sounds good!
> > > >>>
> > > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org>
> > wrote:
> > > >>>
> > > >>>> Hi folks,
> > > >>>>
> > > >>>> I'd like to propose adding a new community provider for DuckDB.
> > > >>>>
> > > >>>> Integration overview:
> > > >>>> ---------------------
> > > >>>>
> > > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike most
> SQL
> > > >>>> providers there is no server to connect to: queries run inside the
> > > task
> > > >>>> process, which makes it a cheap way to run an analytical transform
> > > that
> > > >>>> doesn't justify standing up a cluster. It reads and writes
> Parquet,
> > > CSV
> > > >>> and
> > > >>>> JSON directly from object storage, so it fits naturally between an
> > > >>> extract
> > > >>>> and a load step in a
> > > >>>> Dag.
> > > >>>>
> > > >>>> It also fits Airflow's existing shape well: The proposed DuckDB
> > > >> provider
> > > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather
> than
> > > >>> needing
> > > >>>> new machinery, and it fits in the data-processing/ETL space.
> DuckDB
> > > >> Labs
> > > >>>> was acquired by Amazon but DuckDB remains MIT licensed and
> > controlled
> > > >> by
> > > >>>> the DuckDB Foundation. It is usable with no account, no signup and
> > no
> > > >>>> hosted service.
> > > >>>>
> > > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it works
> > > with
> > > >>>> SQLExecuteQueryOperator and the rest of the common-sql ecosystem,
> > plus
> > > >> a
> > > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> > > connection
> > > >>> is
> > > >>>> optional: with none configured the hook opens an in-memory
> database.
> > > >>>>
> > > >>>> Provider specific integrations would live in their relevant
> provider
> > > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > > >>> apache-airflow-providers-amazon
> > > >>>> for credential management).
> > > >>>>
> > > >>>> System tests:
> > > >>>> -------------
> > > >>>>
> > > >>>> DuckDB runs in-process with no live external service, so system
> > tests
> > > >> are
> > > >>>> not
> > > >>>> required for this provider; the unit tests exercise a real
> in-memory
> > > >>>> DuckDB.
> > > >>>> An AWS-specific system test could be added in the Amazon provider
> > > >>> package,
> > > >>>> if folks think that's required.
> > > >>>>
> > > >>>> Proposed stewards:
> > > >>>> ------------------
> > > >>>>
> > > >>>> Myself and the AWS team plan to sponsor and steward this package,
> > you
> > > >>> know
> > > >>>> us and our track record well :)
> > > >>>>
> > > >>>> - Niko Oliveira (@o-nikolas)
> > > >>>> - Vincent Beck (@vincbeck)
> > > >>>>
> > > >>>> Working implementation
> > > >>>> ----------------------
> > > >>>>
> > > >>>> https://github.com/apache/airflow/pull/72845
> > > >>>>
> > > >>>> Incubation commitment
> > > >>>> ---------------------
> > > >>>>
> > > >>>> We commit to:
> > > >>>>
> > > >>>> - Maintaining the provider and responding to issues within a
> > > reasonable
> > > >>>> time
> > > >>>> - Meeting the incubation health metrics within 6 months
> > > >>>> - Participating in quarterly governance updates
> > > >>>>
> > > >>>> Cheers,
> > > >>>> Niko
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Amogh Desai <am...@apache.org>.
Sounds great, let's bring it in!
Thanks & Regards,
Amogh Desai
On Thu, Sep 10, 2026 at 9:04 PM Niko Oliveira <on...@apache.org> wrote:
> Thanks Julian, much appreciated!
>
> On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
> dev@airflow.apache.org> wrote:
>
> > I wrote our original DuckDB provider at Astronomer, would be more than
> > happy to deprecate in favor of yours! Definitely would love to see
> > something more first-class and better supported.
> >
> > > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <on...@apache.org>
> wrote:
> > >
> > > Glad to see the traction, thanks for weighing in everyone!
> > >
> > >> Is this based on the Astronomer one?
> > > https://github.com/astronomer/airflow-provider-duckdb
> > >
> > > No code was based on that provider. It hasn't been updated in 3 years
> and
> > > it focused on a minimal set of DuckDB features. There wasn't much to
> > reuse
> > > there.
> > >
> > > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> +10
> > >>
> > >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
> > >>
> > >>> Sounds good!
> > >>>
> > >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org>
> wrote:
> > >>>
> > >>>> Hi folks,
> > >>>>
> > >>>> I'd like to propose adding a new community provider for DuckDB.
> > >>>>
> > >>>> Integration overview:
> > >>>> ---------------------
> > >>>>
> > >>>> DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
> > >>>> providers there is no server to connect to: queries run inside the
> > task
> > >>>> process, which makes it a cheap way to run an analytical transform
> > that
> > >>>> doesn't justify standing up a cluster. It reads and writes Parquet,
> > CSV
> > >>> and
> > >>>> JSON directly from object storage, so it fits naturally between an
> > >>> extract
> > >>>> and a load step in a
> > >>>> Dag.
> > >>>>
> > >>>> It also fits Airflow's existing shape well: The proposed DuckDB
> > >> provider
> > >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather than
> > >>> needing
> > >>>> new machinery, and it fits in the data-processing/ETL space. DuckDB
> > >> Labs
> > >>>> was acquired by Amazon but DuckDB remains MIT licensed and
> controlled
> > >> by
> > >>>> the DuckDB Foundation. It is usable with no account, no signup and
> no
> > >>>> hosted service.
> > >>>>
> > >>>> The provider exposes a DuckDBHook extending DbApiHook, so it works
> > with
> > >>>> SQLExecuteQueryOperator and the rest of the common-sql ecosystem,
> plus
> > >> a
> > >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> > connection
> > >>> is
> > >>>> optional: with none configured the hook opens an in-memory database.
> > >>>>
> > >>>> Provider specific integrations would live in their relevant provider
> > >>>> package (e.g. for AWS, an AwsDuckDBHook in
> > >>> apache-airflow-providers-amazon
> > >>>> for credential management).
> > >>>>
> > >>>> System tests:
> > >>>> -------------
> > >>>>
> > >>>> DuckDB runs in-process with no live external service, so system
> tests
> > >> are
> > >>>> not
> > >>>> required for this provider; the unit tests exercise a real in-memory
> > >>>> DuckDB.
> > >>>> An AWS-specific system test could be added in the Amazon provider
> > >>> package,
> > >>>> if folks think that's required.
> > >>>>
> > >>>> Proposed stewards:
> > >>>> ------------------
> > >>>>
> > >>>> Myself and the AWS team plan to sponsor and steward this package,
> you
> > >>> know
> > >>>> us and our track record well :)
> > >>>>
> > >>>> - Niko Oliveira (@o-nikolas)
> > >>>> - Vincent Beck (@vincbeck)
> > >>>>
> > >>>> Working implementation
> > >>>> ----------------------
> > >>>>
> > >>>> https://github.com/apache/airflow/pull/72845
> > >>>>
> > >>>> Incubation commitment
> > >>>> ---------------------
> > >>>>
> > >>>> We commit to:
> > >>>>
> > >>>> - Maintaining the provider and responding to issues within a
> > reasonable
> > >>>> time
> > >>>> - Meeting the incubation health metrics within 6 months
> > >>>> - Participating in quarterly governance updates
> > >>>>
> > >>>> Cheers,
> > >>>> Niko
> > >>>>
> > >>>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Niko Oliveira <on...@apache.org>.
Thanks Julian, much appreciated!
On Thu, Sep 10, 2026 at 8:30 AM Julian LaNeve via dev <
dev@airflow.apache.org> wrote:
> I wrote our original DuckDB provider at Astronomer, would be more than
> happy to deprecate in favor of yours! Definitely would love to see
> something more first-class and better supported.
>
> > On Sep 10, 2026, at 11:23 AM, Niko Oliveira <on...@apache.org> wrote:
> >
> > Glad to see the traction, thanks for weighing in everyone!
> >
> >> Is this based on the Astronomer one?
> > https://github.com/astronomer/airflow-provider-duckdb
> >
> > No code was based on that provider. It hasn't been updated in 3 years and
> > it focused on a minimal set of DuckDB features. There wasn't much to
> reuse
> > there.
> >
> > On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> +10
> >>
> >> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
> >>
> >>> Sounds good!
> >>>
> >>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org> wrote:
> >>>
> >>>> Hi folks,
> >>>>
> >>>> I'd like to propose adding a new community provider for DuckDB.
> >>>>
> >>>> Integration overview:
> >>>> ---------------------
> >>>>
> >>>> DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
> >>>> providers there is no server to connect to: queries run inside the
> task
> >>>> process, which makes it a cheap way to run an analytical transform
> that
> >>>> doesn't justify standing up a cluster. It reads and writes Parquet,
> CSV
> >>> and
> >>>> JSON directly from object storage, so it fits naturally between an
> >>> extract
> >>>> and a load step in a
> >>>> Dag.
> >>>>
> >>>> It also fits Airflow's existing shape well: The proposed DuckDB
> >> provider
> >>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather than
> >>> needing
> >>>> new machinery, and it fits in the data-processing/ETL space. DuckDB
> >> Labs
> >>>> was acquired by Amazon but DuckDB remains MIT licensed and controlled
> >> by
> >>>> the DuckDB Foundation. It is usable with no account, no signup and no
> >>>> hosted service.
> >>>>
> >>>> The provider exposes a DuckDBHook extending DbApiHook, so it works
> with
> >>>> SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus
> >> a
> >>>> DuckDBExecuteQueryOperator and a duckdb connection type. The
> connection
> >>> is
> >>>> optional: with none configured the hook opens an in-memory database.
> >>>>
> >>>> Provider specific integrations would live in their relevant provider
> >>>> package (e.g. for AWS, an AwsDuckDBHook in
> >>> apache-airflow-providers-amazon
> >>>> for credential management).
> >>>>
> >>>> System tests:
> >>>> -------------
> >>>>
> >>>> DuckDB runs in-process with no live external service, so system tests
> >> are
> >>>> not
> >>>> required for this provider; the unit tests exercise a real in-memory
> >>>> DuckDB.
> >>>> An AWS-specific system test could be added in the Amazon provider
> >>> package,
> >>>> if folks think that's required.
> >>>>
> >>>> Proposed stewards:
> >>>> ------------------
> >>>>
> >>>> Myself and the AWS team plan to sponsor and steward this package, you
> >>> know
> >>>> us and our track record well :)
> >>>>
> >>>> - Niko Oliveira (@o-nikolas)
> >>>> - Vincent Beck (@vincbeck)
> >>>>
> >>>> Working implementation
> >>>> ----------------------
> >>>>
> >>>> https://github.com/apache/airflow/pull/72845
> >>>>
> >>>> Incubation commitment
> >>>> ---------------------
> >>>>
> >>>> We commit to:
> >>>>
> >>>> - Maintaining the provider and responding to issues within a
> reasonable
> >>>> time
> >>>> - Meeting the incubation health metrics within 6 months
> >>>> - Participating in quarterly governance updates
> >>>>
> >>>> Cheers,
> >>>> Niko
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Julian LaNeve via dev <de...@airflow.apache.org>.
I wrote our original DuckDB provider at Astronomer, would be more than happy to deprecate in favor of yours! Definitely would love to see something more first-class and better supported.
> On Sep 10, 2026, at 11:23 AM, Niko Oliveira <on...@apache.org> wrote:
>
> Glad to see the traction, thanks for weighing in everyone!
>
>> Is this based on the Astronomer one?
> https://github.com/astronomer/airflow-provider-duckdb
>
> No code was based on that provider. It hasn't been updated in 3 years and
> it focused on a minimal set of DuckDB features. There wasn't much to reuse
> there.
>
> On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> +10
>>
>> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
>>
>>> Sounds good!
>>>
>>> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> I'd like to propose adding a new community provider for DuckDB.
>>>>
>>>> Integration overview:
>>>> ---------------------
>>>>
>>>> DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
>>>> providers there is no server to connect to: queries run inside the task
>>>> process, which makes it a cheap way to run an analytical transform that
>>>> doesn't justify standing up a cluster. It reads and writes Parquet, CSV
>>> and
>>>> JSON directly from object storage, so it fits naturally between an
>>> extract
>>>> and a load step in a
>>>> Dag.
>>>>
>>>> It also fits Airflow's existing shape well: The proposed DuckDB
>> provider
>>>> speaks DbAPI, so it plugs into the common-sql ecosystem rather than
>>> needing
>>>> new machinery, and it fits in the data-processing/ETL space. DuckDB
>> Labs
>>>> was acquired by Amazon but DuckDB remains MIT licensed and controlled
>> by
>>>> the DuckDB Foundation. It is usable with no account, no signup and no
>>>> hosted service.
>>>>
>>>> The provider exposes a DuckDBHook extending DbApiHook, so it works with
>>>> SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus
>> a
>>>> DuckDBExecuteQueryOperator and a duckdb connection type. The connection
>>> is
>>>> optional: with none configured the hook opens an in-memory database.
>>>>
>>>> Provider specific integrations would live in their relevant provider
>>>> package (e.g. for AWS, an AwsDuckDBHook in
>>> apache-airflow-providers-amazon
>>>> for credential management).
>>>>
>>>> System tests:
>>>> -------------
>>>>
>>>> DuckDB runs in-process with no live external service, so system tests
>> are
>>>> not
>>>> required for this provider; the unit tests exercise a real in-memory
>>>> DuckDB.
>>>> An AWS-specific system test could be added in the Amazon provider
>>> package,
>>>> if folks think that's required.
>>>>
>>>> Proposed stewards:
>>>> ------------------
>>>>
>>>> Myself and the AWS team plan to sponsor and steward this package, you
>>> know
>>>> us and our track record well :)
>>>>
>>>> - Niko Oliveira (@o-nikolas)
>>>> - Vincent Beck (@vincbeck)
>>>>
>>>> Working implementation
>>>> ----------------------
>>>>
>>>> https://github.com/apache/airflow/pull/72845
>>>>
>>>> Incubation commitment
>>>> ---------------------
>>>>
>>>> We commit to:
>>>>
>>>> - Maintaining the provider and responding to issues within a reasonable
>>>> time
>>>> - Meeting the incubation health metrics within 6 months
>>>> - Participating in quarterly governance updates
>>>>
>>>> Cheers,
>>>> Niko
>>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Niko Oliveira <on...@apache.org>.
Glad to see the traction, thanks for weighing in everyone!
> Is this based on the Astronomer one?
https://github.com/astronomer/airflow-provider-duckdb
No code was based on that provider. It hasn't been updated in 3 years and
it focused on a minimal set of DuckDB features. There wasn't much to reuse
there.
On Thu, Sep 10, 2026 at 1:41 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> +10
>
> On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
>
> > Sounds good!
> >
> > On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org> wrote:
> >
> > > Hi folks,
> > >
> > > I'd like to propose adding a new community provider for DuckDB.
> > >
> > > Integration overview:
> > > ---------------------
> > >
> > > DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
> > > providers there is no server to connect to: queries run inside the task
> > > process, which makes it a cheap way to run an analytical transform that
> > > doesn't justify standing up a cluster. It reads and writes Parquet, CSV
> > and
> > > JSON directly from object storage, so it fits naturally between an
> > extract
> > > and a load step in a
> > > Dag.
> > >
> > > It also fits Airflow's existing shape well: The proposed DuckDB
> provider
> > > speaks DbAPI, so it plugs into the common-sql ecosystem rather than
> > needing
> > > new machinery, and it fits in the data-processing/ETL space. DuckDB
> Labs
> > > was acquired by Amazon but DuckDB remains MIT licensed and controlled
> by
> > > the DuckDB Foundation. It is usable with no account, no signup and no
> > > hosted service.
> > >
> > > The provider exposes a DuckDBHook extending DbApiHook, so it works with
> > > SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus
> a
> > > DuckDBExecuteQueryOperator and a duckdb connection type. The connection
> > is
> > > optional: with none configured the hook opens an in-memory database.
> > >
> > > Provider specific integrations would live in their relevant provider
> > > package (e.g. for AWS, an AwsDuckDBHook in
> > apache-airflow-providers-amazon
> > > for credential management).
> > >
> > > System tests:
> > > -------------
> > >
> > > DuckDB runs in-process with no live external service, so system tests
> are
> > > not
> > > required for this provider; the unit tests exercise a real in-memory
> > > DuckDB.
> > > An AWS-specific system test could be added in the Amazon provider
> > package,
> > > if folks think that's required.
> > >
> > > Proposed stewards:
> > > ------------------
> > >
> > > Myself and the AWS team plan to sponsor and steward this package, you
> > know
> > > us and our track record well :)
> > >
> > > - Niko Oliveira (@o-nikolas)
> > > - Vincent Beck (@vincbeck)
> > >
> > > Working implementation
> > > ----------------------
> > >
> > > https://github.com/apache/airflow/pull/72845
> > >
> > > Incubation commitment
> > > ---------------------
> > >
> > > We commit to:
> > >
> > > - Maintaining the provider and responding to issues within a reasonable
> > > time
> > > - Meeting the incubation health metrics within 6 months
> > > - Participating in quarterly governance updates
> > >
> > > Cheers,
> > > Niko
> > >
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Jarek Potiuk <ja...@potiuk.com>.
+10
On Thu, Sep 10, 2026 at 8:40 AM Dev-iL <gi...@gmail.com> wrote:
> Sounds good!
>
> On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org> wrote:
>
> > Hi folks,
> >
> > I'd like to propose adding a new community provider for DuckDB.
> >
> > Integration overview:
> > ---------------------
> >
> > DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
> > providers there is no server to connect to: queries run inside the task
> > process, which makes it a cheap way to run an analytical transform that
> > doesn't justify standing up a cluster. It reads and writes Parquet, CSV
> and
> > JSON directly from object storage, so it fits naturally between an
> extract
> > and a load step in a
> > Dag.
> >
> > It also fits Airflow's existing shape well: The proposed DuckDB provider
> > speaks DbAPI, so it plugs into the common-sql ecosystem rather than
> needing
> > new machinery, and it fits in the data-processing/ETL space. DuckDB Labs
> > was acquired by Amazon but DuckDB remains MIT licensed and controlled by
> > the DuckDB Foundation. It is usable with no account, no signup and no
> > hosted service.
> >
> > The provider exposes a DuckDBHook extending DbApiHook, so it works with
> > SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a
> > DuckDBExecuteQueryOperator and a duckdb connection type. The connection
> is
> > optional: with none configured the hook opens an in-memory database.
> >
> > Provider specific integrations would live in their relevant provider
> > package (e.g. for AWS, an AwsDuckDBHook in
> apache-airflow-providers-amazon
> > for credential management).
> >
> > System tests:
> > -------------
> >
> > DuckDB runs in-process with no live external service, so system tests are
> > not
> > required for this provider; the unit tests exercise a real in-memory
> > DuckDB.
> > An AWS-specific system test could be added in the Amazon provider
> package,
> > if folks think that's required.
> >
> > Proposed stewards:
> > ------------------
> >
> > Myself and the AWS team plan to sponsor and steward this package, you
> know
> > us and our track record well :)
> >
> > - Niko Oliveira (@o-nikolas)
> > - Vincent Beck (@vincbeck)
> >
> > Working implementation
> > ----------------------
> >
> > https://github.com/apache/airflow/pull/72845
> >
> > Incubation commitment
> > ---------------------
> >
> > We commit to:
> >
> > - Maintaining the provider and responding to issues within a reasonable
> > time
> > - Meeting the incubation health metrics within 6 months
> > - Participating in quarterly governance updates
> >
> > Cheers,
> > Niko
> >
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Dev-iL <gi...@gmail.com>.
Sounds good!
On Thu, 10 Sept 2026, 9:04 Niko Oliveira, <on...@apache.org> wrote:
> Hi folks,
>
> I'd like to propose adding a new community provider for DuckDB.
>
> Integration overview:
> ---------------------
>
> DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
> providers there is no server to connect to: queries run inside the task
> process, which makes it a cheap way to run an analytical transform that
> doesn't justify standing up a cluster. It reads and writes Parquet, CSV and
> JSON directly from object storage, so it fits naturally between an extract
> and a load step in a
> Dag.
>
> It also fits Airflow's existing shape well: The proposed DuckDB provider
> speaks DbAPI, so it plugs into the common-sql ecosystem rather than needing
> new machinery, and it fits in the data-processing/ETL space. DuckDB Labs
> was acquired by Amazon but DuckDB remains MIT licensed and controlled by
> the DuckDB Foundation. It is usable with no account, no signup and no
> hosted service.
>
> The provider exposes a DuckDBHook extending DbApiHook, so it works with
> SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a
> DuckDBExecuteQueryOperator and a duckdb connection type. The connection is
> optional: with none configured the hook opens an in-memory database.
>
> Provider specific integrations would live in their relevant provider
> package (e.g. for AWS, an AwsDuckDBHook in apache-airflow-providers-amazon
> for credential management).
>
> System tests:
> -------------
>
> DuckDB runs in-process with no live external service, so system tests are
> not
> required for this provider; the unit tests exercise a real in-memory
> DuckDB.
> An AWS-specific system test could be added in the Amazon provider package,
> if folks think that's required.
>
> Proposed stewards:
> ------------------
>
> Myself and the AWS team plan to sponsor and steward this package, you know
> us and our track record well :)
>
> - Niko Oliveira (@o-nikolas)
> - Vincent Beck (@vincbeck)
>
> Working implementation
> ----------------------
>
> https://github.com/apache/airflow/pull/72845
>
> Incubation commitment
> ---------------------
>
> We commit to:
>
> - Maintaining the provider and responding to issues within a reasonable
> time
> - Meeting the incubation health metrics within 6 months
> - Participating in quarterly governance updates
>
> Cheers,
> Niko
>
Re: [DISCUSS] New Community Provider: DuckDB
Posted by Blain David <da...@infrabel.be>.
This is a good initiative, I'm in favor for this one!
Is this based on the Astronomer one? https://github.com/astronomer/airflow-provider-duckdb
Kind regards,
David
General (Internal Property)
________________________________
From: Niko Oliveira <on...@apache.org>
Sent: Thursday, September 10, 2026 08:03
To: dev@airflow.apache.org <de...@airflow.apache.org>
Subject: [DISCUSS] New Community Provider: DuckDB
EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar abuse@infrabel.be<ma...@infrabel.be>.
Hi folks,
I'd like to propose adding a new community provider for DuckDB.
Integration overview:
---------------------
DuckDB is an in-process analytical (OLAP) database. Unlike most SQL
providers there is no server to connect to: queries run inside the task
process, which makes it a cheap way to run an analytical transform that
doesn't justify standing up a cluster. It reads and writes Parquet, CSV and
JSON directly from object storage, so it fits naturally between an extract
and a load step in a
Dag.
It also fits Airflow's existing shape well: The proposed DuckDB provider
speaks DbAPI, so it plugs into the common-sql ecosystem rather than needing
new machinery, and it fits in the data-processing/ETL space. DuckDB Labs
was acquired by Amazon but DuckDB remains MIT licensed and controlled by
the DuckDB Foundation. It is usable with no account, no signup and no
hosted service.
The provider exposes a DuckDBHook extending DbApiHook, so it works with
SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a
DuckDBExecuteQueryOperator and a duckdb connection type. The connection is
optional: with none configured the hook opens an in-memory database.
Provider specific integrations would live in their relevant provider
package (e.g. for AWS, an AwsDuckDBHook in apache-airflow-providers-amazon
for credential management).
System tests:
-------------
DuckDB runs in-process with no live external service, so system tests are
not
required for this provider; the unit tests exercise a real in-memory DuckDB.
An AWS-specific system test could be added in the Amazon provider package,
if folks think that's required.
Proposed stewards:
------------------
Myself and the AWS team plan to sponsor and steward this package, you know
us and our track record well :)
- Niko Oliveira (@o-nikolas)
- Vincent Beck (@vincbeck)
Working implementation
----------------------
https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F72845&data=05%7C02%7Cdavid.blain%40infrabel.be%7C8a280eeef1984f1ea80808df0f015e29%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639246170692897451%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Rc9w5Ilzv4w0Ki1kpwMyP6sNAR0no5m7OtN1WdwSt24%3D&reserved=0<https://github.com/apache/airflow/pull/72845>
Incubation commitment
---------------------
We commit to:
- Maintaining the provider and responding to issues within a reasonable time
- Meeting the incubation health metrics within 6 months
- Participating in quarterly governance updates
Cheers,
Niko