You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by David Capwell <dc...@gmail.com> on 2017/06/28 18:05:21 UTC
Hooks and connection
I'm just starting out with airflow and looking to add my own artifactory
hook so my tasks can pull from there.
Looking at the docs this means I need a ArtifactoryHook but not clear to me
how this integrates with connections. Looking over the connection code the
mapping is hard coded but the plug-in docs say to just build a hook. I
have a hello world hook but not sure how to link it with connections
(connection tab does not list it)
Thanks for your time reading this email
Re: Hooks and connection
Posted by Gael Magnan <ga...@gmail.com>.
No problem, here it is:
https://github.com/apache/incubator-airflow/pull/1981 , it's not up to date
and probably could use a lot of work still, but it was a start.
Regards
Le jeu. 29 juin 2017 à 21:05, Rutherford, James <
James.Rutherford@maplecroft.com> a écrit :
> Hi Gael,
>
> This sounds interesting, but I couldn’t find the PR. Could you link to it?
>
> Cheers,
>
> Jim
>
>
>
>
> On 28/06/2017, 22:08, "Gael Magnan" <ga...@gmail.com> wrote:
>
> >Hi,
> >
> >A few months ago I started working on a refactoring of the connections so
> >it would be easier to add new types. It never got merged but you probably
> >can find it still in the pr liste.
> >
> >I think a complete overhaul of the connections is more than needed.
> >Especially removing the default ones and the hard coded types.
> >
> >Regards
> >Gael
> >
> >Le mer. 28 juin 2017 à 21:39, David Capwell <dc...@gmail.com> a écrit
> :
> >
> >> Thanks! Yeah, using http type for now and things are working just fine
> >>
> >> On Jun 28, 2017 12:17 PM, "Rutherford, James" <
> >> James.Rutherford@maplecroft.com> wrote:
> >>
> >> > Hi David,
> >> >
> >> > I’m also a bit of an Airflow novice but what I’ve figured out is that
> >> > Connections are essentially parameters for Hooks. The connection type
> >> > doesn't seem to be used for anything explicitly (the list of choices
> is
> >> > hard-coded in the Airflow source).
> >> >
> >> > For example, I made a SharePoint hook that defines how content goes
> into
> >> > and comes out of SharePoint. The connection information (URL,
> protocol,
> >> > credentials, etc) is stored as a Connection with type set as 'Samba'
> (it
> >> > was the closest I could find); the Hook just wraps the SharePoint API.
> >> >
> >> >
> >> > I hope that helps.
> >> >
> >> > Cheers,
> >> >
> >> > Jim
> >> >
> >> >
> >> >
> >> > On 28/06/2017, 19:05, "David Capwell" <dc...@gmail.com> wrote:
> >> >
> >> > >I'm just starting out with airflow and looking to add my own
> artifactory
> >> > >hook so my tasks can pull from there.
> >> > >
> >> > >Looking at the docs this means I need a ArtifactoryHook but not
> clear to
> >> > me
> >> > >how this integrates with connections. Looking over the connection
> code
> >> > the
> >> > >mapping is hard coded but the plug-in docs say to just build a
> hook. I
> >> > >have a hello world hook but not sure how to link it with connections
> >> > >(connection tab does not list it)
> >> > >
> >> > >Thanks for your time reading this email
> >> >
> >> > ________________________________
> >> >
> >> > This email is intended solely for the recipient. It may contain
> >> > privileged, proprietary or confidential information or material. If
> you
> >> are
> >> > not the intended recipient, please delete this email and any
> attachments
> >> > and notify the sender of the error.
> >> >
> >>
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If you are
> not the intended recipient, please delete this email and any attachments
> and notify the sender of the error.
>
Re: Hooks and connection
Posted by "Rutherford, James" <Ja...@maplecroft.com>.
Hi Gael,
This sounds interesting, but I couldn’t find the PR. Could you link to it?
Cheers,
Jim
On 28/06/2017, 22:08, "Gael Magnan" <ga...@gmail.com> wrote:
>Hi,
>
>A few months ago I started working on a refactoring of the connections so
>it would be easier to add new types. It never got merged but you probably
>can find it still in the pr liste.
>
>I think a complete overhaul of the connections is more than needed.
>Especially removing the default ones and the hard coded types.
>
>Regards
>Gael
>
>Le mer. 28 juin 2017 à 21:39, David Capwell <dc...@gmail.com> a écrit :
>
>> Thanks! Yeah, using http type for now and things are working just fine
>>
>> On Jun 28, 2017 12:17 PM, "Rutherford, James" <
>> James.Rutherford@maplecroft.com> wrote:
>>
>> > Hi David,
>> >
>> > I’m also a bit of an Airflow novice but what I’ve figured out is that
>> > Connections are essentially parameters for Hooks. The connection type
>> > doesn't seem to be used for anything explicitly (the list of choices is
>> > hard-coded in the Airflow source).
>> >
>> > For example, I made a SharePoint hook that defines how content goes into
>> > and comes out of SharePoint. The connection information (URL, protocol,
>> > credentials, etc) is stored as a Connection with type set as 'Samba' (it
>> > was the closest I could find); the Hook just wraps the SharePoint API.
>> >
>> >
>> > I hope that helps.
>> >
>> > Cheers,
>> >
>> > Jim
>> >
>> >
>> >
>> > On 28/06/2017, 19:05, "David Capwell" <dc...@gmail.com> wrote:
>> >
>> > >I'm just starting out with airflow and looking to add my own artifactory
>> > >hook so my tasks can pull from there.
>> > >
>> > >Looking at the docs this means I need a ArtifactoryHook but not clear to
>> > me
>> > >how this integrates with connections. Looking over the connection code
>> > the
>> > >mapping is hard coded but the plug-in docs say to just build a hook. I
>> > >have a hello world hook but not sure how to link it with connections
>> > >(connection tab does not list it)
>> > >
>> > >Thanks for your time reading this email
>> >
>> > ________________________________
>> >
>> > This email is intended solely for the recipient. It may contain
>> > privileged, proprietary or confidential information or material. If you
>> are
>> > not the intended recipient, please delete this email and any attachments
>> > and notify the sender of the error.
>> >
>>
________________________________
This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.
Re: Hooks and connection
Posted by Gael Magnan <ga...@gmail.com>.
Hi,
A few months ago I started working on a refactoring of the connections so
it would be easier to add new types. It never got merged but you probably
can find it still in the pr liste.
I think a complete overhaul of the connections is more than needed.
Especially removing the default ones and the hard coded types.
Regards
Gael
Le mer. 28 juin 2017 à 21:39, David Capwell <dc...@gmail.com> a écrit :
> Thanks! Yeah, using http type for now and things are working just fine
>
> On Jun 28, 2017 12:17 PM, "Rutherford, James" <
> James.Rutherford@maplecroft.com> wrote:
>
> > Hi David,
> >
> > I’m also a bit of an Airflow novice but what I’ve figured out is that
> > Connections are essentially parameters for Hooks. The connection type
> > doesn't seem to be used for anything explicitly (the list of choices is
> > hard-coded in the Airflow source).
> >
> > For example, I made a SharePoint hook that defines how content goes into
> > and comes out of SharePoint. The connection information (URL, protocol,
> > credentials, etc) is stored as a Connection with type set as 'Samba' (it
> > was the closest I could find); the Hook just wraps the SharePoint API.
> >
> >
> > I hope that helps.
> >
> > Cheers,
> >
> > Jim
> >
> >
> >
> > On 28/06/2017, 19:05, "David Capwell" <dc...@gmail.com> wrote:
> >
> > >I'm just starting out with airflow and looking to add my own artifactory
> > >hook so my tasks can pull from there.
> > >
> > >Looking at the docs this means I need a ArtifactoryHook but not clear to
> > me
> > >how this integrates with connections. Looking over the connection code
> > the
> > >mapping is hard coded but the plug-in docs say to just build a hook. I
> > >have a hello world hook but not sure how to link it with connections
> > >(connection tab does not list it)
> > >
> > >Thanks for your time reading this email
> >
> > ________________________________
> >
> > This email is intended solely for the recipient. It may contain
> > privileged, proprietary or confidential information or material. If you
> are
> > not the intended recipient, please delete this email and any attachments
> > and notify the sender of the error.
> >
>
Re: Hooks and connection
Posted by David Capwell <dc...@gmail.com>.
Thanks! Yeah, using http type for now and things are working just fine
On Jun 28, 2017 12:17 PM, "Rutherford, James" <
James.Rutherford@maplecroft.com> wrote:
> Hi David,
>
> I’m also a bit of an Airflow novice but what I’ve figured out is that
> Connections are essentially parameters for Hooks. The connection type
> doesn't seem to be used for anything explicitly (the list of choices is
> hard-coded in the Airflow source).
>
> For example, I made a SharePoint hook that defines how content goes into
> and comes out of SharePoint. The connection information (URL, protocol,
> credentials, etc) is stored as a Connection with type set as 'Samba' (it
> was the closest I could find); the Hook just wraps the SharePoint API.
>
>
> I hope that helps.
>
> Cheers,
>
> Jim
>
>
>
> On 28/06/2017, 19:05, "David Capwell" <dc...@gmail.com> wrote:
>
> >I'm just starting out with airflow and looking to add my own artifactory
> >hook so my tasks can pull from there.
> >
> >Looking at the docs this means I need a ArtifactoryHook but not clear to
> me
> >how this integrates with connections. Looking over the connection code
> the
> >mapping is hard coded but the plug-in docs say to just build a hook. I
> >have a hello world hook but not sure how to link it with connections
> >(connection tab does not list it)
> >
> >Thanks for your time reading this email
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain
> privileged, proprietary or confidential information or material. If you are
> not the intended recipient, please delete this email and any attachments
> and notify the sender of the error.
>
Re: Hooks and connection
Posted by "Rutherford, James" <Ja...@maplecroft.com>.
Hi David,
I’m also a bit of an Airflow novice but what I’ve figured out is that
Connections are essentially parameters for Hooks. The connection type
doesn't seem to be used for anything explicitly (the list of choices is
hard-coded in the Airflow source).
For example, I made a SharePoint hook that defines how content goes into
and comes out of SharePoint. The connection information (URL, protocol,
credentials, etc) is stored as a Connection with type set as 'Samba' (it
was the closest I could find); the Hook just wraps the SharePoint API.
I hope that helps.
Cheers,
Jim
On 28/06/2017, 19:05, "David Capwell" <dc...@gmail.com> wrote:
>I'm just starting out with airflow and looking to add my own artifactory
>hook so my tasks can pull from there.
>
>Looking at the docs this means I need a ArtifactoryHook but not clear to me
>how this integrates with connections. Looking over the connection code the
>mapping is hard coded but the plug-in docs say to just build a hook. I
>have a hello world hook but not sure how to link it with connections
>(connection tab does not list it)
>
>Thanks for your time reading this email
________________________________
This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.
Re: Hooks and connection
Posted by Maxime Beauchemin <ma...@gmail.com>.
From your hook you can call `self.get_connection("foo_conn_id")` (a static
method of BaseHook) to get a connection object.
It's also common practice for hooks to implement `get_conn` (don't confuse
it with BaseHook's `get_connection`) which is expected to return an active,
already connected client or connection object.
Max
On Wed, Jun 28, 2017 at 11:05 AM, David Capwell <dc...@gmail.com> wrote:
> I'm just starting out with airflow and looking to add my own artifactory
> hook so my tasks can pull from there.
>
> Looking at the docs this means I need a ArtifactoryHook but not clear to me
> how this integrates with connections. Looking over the connection code the
> mapping is hard coded but the plug-in docs say to just build a hook. I
> have a hello world hook but not sure how to link it with connections
> (connection tab does not list it)
>
> Thanks for your time reading this email
>