You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ryan Hatter <ry...@gmail.com> on 2021/03/03 18:04:01 UTC
dbt Provider
Hey all,
dbt <https://docs.getdbt.com/docs/introduction/> seems to continue to gain
momentum. There's already an airflow-dbt project
<https://github.com/gocardless/airflow-dbt> that is essentially a provider
package. Would it make sense to fold the dbt_hook
<https://github.com/gocardless/airflow-dbt/blob/master/airflow_dbt/hooks/dbt_hook.py>
and dbt_operator
<https://github.com/gocardless/airflow-dbt/blob/master/airflow_dbt/operators/dbt_operator.py>
into a provider package in the official airflow repo?
Re: dbt Provider
Posted by Ryan Hatter <ry...@gmail.com>.
Hey Pete!
That blog is actually what piqued my interest. It seems like most of the heavy lifting is done. I’m happy to help if I can.
I’ll reach out on slack :)
> On Mar 5, 2021, at 10:26, Peter DeJoy <pd...@gmail.com> wrote:
>
>
> Hey Ryan,
>
> Excited that you’re interested in this- as it turns out, we @ Astronomer have done some initial work with the dbt team to build an official provider package for dbt and dbt Cloud. We wrote this blog series a couple of months back, but we’re really hoping to evolve the integration between Airflow and dbt so that running dbt models from your Airflow tasks is supported first-party by a provider and its subsequent modules.
>
> More on that soon, but happy to chat with you and anyone else in the community about the user patterns you’d like to have supported. Feel free to get in touch.
>
> Pete
>> On Mar 4, 2021, 9:30 AM -0500, Ryan Hatter <ry...@gmail.com>, wrote:
>> Would it be appropriate for me to reach out to one of the airflow-dbt maintainers and see if they’d be interested in managing a provider?
>>
>>> On Mar 3, 2021, at 13:23, Ash Berlin-Taylor <as...@apache.org> wrote:
>>>
>>> I honestly think no: not because it's not useful, but because dbt have the time and ability to maintain their own provider package, much like Great Expectations do:
>>>
>>> https://greatexpectations.io/blog/airflow-operator/
>>>
>>> I'd much rather we work with dbt to do what ever is needed to make it a full provider than pull it in tree when it already exists as a third party package.
>>>
>>> -ash
>>>
>>>> On 3 March 2021 18:04:01 GMT, Ryan Hatter <ry...@gmail.com> wrote:
>>>> Hey all,
>>>>
>>>> dbt seems to continue to gain momentum. There's already an airflow-dbt project that is essentially a provider package. Would it make sense to fold the dbt_hook and dbt_operator into a provider package in the official airflow repo?
Re: dbt Provider
Posted by Peter DeJoy <pd...@gmail.com>.
Hey Ryan,
Excited that you’re interested in this- as it turns out, we @ Astronomer have done some initial work with the dbt team to build an official provider package for dbt and dbt Cloud. We wrote this blog series a couple of months back, but we’re really hoping to evolve the integration between Airflow and dbt so that running dbt models from your Airflow tasks is supported first-party by a provider and its subsequent modules.
More on that soon, but happy to chat with you and anyone else in the community about the user patterns you’d like to have supported. Feel free to get in touch.
Pete
On Mar 4, 2021, 9:30 AM -0500, Ryan Hatter <ry...@gmail.com>, wrote:
> Would it be appropriate for me to reach out to one of the airflow-dbt maintainers and see if they’d be interested in managing a provider?
>
> > On Mar 3, 2021, at 13:23, Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > I honestly think no: not because it's not useful, but because dbt have the time and ability to maintain their own provider package, much like Great Expectations do:
> >
> > https://greatexpectations.io/blog/airflow-operator/
> >
> > I'd much rather we work with dbt to do what ever is needed to make it a full provider than pull it in tree when it already exists as a third party package.
> >
> > -ash
> >
> > > On 3 March 2021 18:04:01 GMT, Ryan Hatter <ry...@gmail.com> wrote:
> > > > Hey all,
> > > >
> > > > dbt seems to continue to gain momentum. There's already an airflow-dbt project that is essentially a provider package. Would it make sense to fold the dbt_hook and dbt_operator into a provider package in the official airflow repo?
Re: dbt Provider
Posted by Ryan Hatter <ry...@gmail.com>.
Would it be appropriate for me to reach out to one of the airflow-dbt maintainers and see if they’d be interested in managing a provider?
> On Mar 3, 2021, at 13:23, Ash Berlin-Taylor <as...@apache.org> wrote:
>
> I honestly think no: not because it's not useful, but because dbt have the time and ability to maintain their own provider package, much like Great Expectations do:
>
> https://greatexpectations.io/blog/airflow-operator/
>
> I'd much rather we work with dbt to do what ever is needed to make it a full provider than pull it in tree when it already exists as a third party package.
>
> -ash
>
>> On 3 March 2021 18:04:01 GMT, Ryan Hatter <ry...@gmail.com> wrote:
>> Hey all,
>>
>> dbt seems to continue to gain momentum. There's already an airflow-dbt project that is essentially a provider package. Would it make sense to fold the dbt_hook and dbt_operator into a provider package in the official airflow repo?
Re: dbt Provider
Posted by Ash Berlin-Taylor <as...@apache.org>.
I honestly think no: not because it's not useful, but because dbt have the time and ability to maintain their own provider package, much like Great Expectations do:
https://greatexpectations.io/blog/airflow-operator/
I'd much rather we work with dbt to do what ever is needed to make it a full provider than pull it in tree when it already exists as a third party package.
-ash
On 3 March 2021 18:04:01 GMT, Ryan Hatter <ry...@gmail.com> wrote:
>Hey all,
>
>dbt <https://docs.getdbt.com/docs/introduction/> seems to continue to gain
>momentum. There's already an airflow-dbt project
><https://github.com/gocardless/airflow-dbt> that is essentially a provider
>package. Would it make sense to fold the dbt_hook
><https://github.com/gocardless/airflow-dbt/blob/master/airflow_dbt/hooks/dbt_hook.py>
>and dbt_operator
><https://github.com/gocardless/airflow-dbt/blob/master/airflow_dbt/operators/dbt_operator.py>
>into a provider package in the official airflow repo?