You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ben Laird <br...@gmail.com> on 2018/08/07 13:38:47 UTC
Multiple hosts for a single connection
Hello -
I'd like to define multiple connections for the `webhdfs_default` conn_id,
as we have a primary and backup host for the hadoop namenode. This is for
use with the WebHDFSHook, which queries for connections and seemingly can
accept a list of them from the database.
```
nn_connections = self.get_connections(self.webhdfs_conn_id)
for nn in nn_connections:
```
https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55
I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
each hostname, however with the CLI and adding the entries to the
airflow.cfg I get errors that a connection already exists.
It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
Is this a bug? Or is there a different way to accomplish without manually
adding via the UI?
Thanks!
Ben Laird
Re: Multiple hosts for a single connection
Posted by Ben Laird <br...@gmail.com>.
Hi Ash -
That'd be perfect. Since conn_id isn't a primary key (since we can add via
the UI), I agree that it'd be great to have the same functionality from the
CLI with a flag. I can hopefully do this and submit a PR if folks would
find it useful!
Ben
On Tue, Aug 7, 2018 at 10:28 AM, Ash Berlin-Taylor <as...@apache.org> wrote:
> Hmm yes, it appears that the `airflow connections` CLI doesn't let you
> create multiple connections of the same conn_id. What the WebUI can do the
> CLI should be able to do also! It should allow that in some way (behind a
> `--allow-multiple` flag perhaps? I can see an argument for not allowing set
> multiple by default as it is often not what people want.)
>
> -ash
>
> > On 7 Aug 2018, at 15:17, Deng Xiaodong <xd...@gmail.com> wrote:
> >
> > Hi Ben,
> >
> > If you would like to set multiple connections with the same *conn_id*,
> you
> > can only do that in Web UI. That is, you need to build multiple entries
> > with different host, but with the same *conn_id*.
> >
> > The method of setting connection in environment variables can only help
> set
> > one connection for each *conn_id*.
> >
> > (Of course Web UI stores your connections in the metadata database. So if
> > you really need to avoid manual interventions and need multiple
> connections
> > for one single conn_id, hacking into the metadata database may be an
> > option? even thought not recommended)
> >
> > Other guys please correct me if I'm wrong in any parts. Thanks.
> >
> >
> > Regards,
> > XD
> >
> > On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br...@gmail.com> wrote:
> >
> >> Hello -
> >>
> >> I'd like to define multiple connections for the `webhdfs_default`
> conn_id,
> >> as we have a primary and backup host for the hadoop namenode. This is
> for
> >> use with the WebHDFSHook, which queries for connections and seemingly
> can
> >> accept a list of them from the database.
> >>
> >> ```
> >> nn_connections = self.get_connections(self.webhdfs_conn_id)
> >> for nn in nn_connections:
> >> ```
> >>
> >> https://github.com/apache/incubator-airflow/blob/master/
> airflow/hooks/webhdfs_hook.py#L54-L55
> >>
> >> I can manually add 2 entries for `webhdfs_default` in the Web UI, one
> for
> >> each hostname, however with the CLI and adding the entries to the
> >> airflow.cfg I get errors that a connection already exists.
> >>
> >> It seems WebHDFSHook doesn't support reading multiple hosts from one
> entry.
> >> Is this a bug? Or is there a different way to accomplish without
> manually
> >> adding via the UI?
> >>
> >> Thanks!
> >> Ben Laird
> >>
>
>
Re: Multiple hosts for a single connection
Posted by Naik Kaxil <k....@reply.com>.
Agree that we should have the same functionality over the WebUI and CLI. We need multiple connections for a single conn-id for load balancing, so +1 for adding something like `--allow-multiple` or just a warning when a connection with the same id is created with cli.
On 07/08/2018, 15:29, "Ash Berlin-Taylor" <as...@apache.org> wrote:
Hmm yes, it appears that the `airflow connections` CLI doesn't let you create multiple connections of the same conn_id. What the WebUI can do the CLI should be able to do also! It should allow that in some way (behind a `--allow-multiple` flag perhaps? I can see an argument for not allowing set multiple by default as it is often not what people want.)
-ash
> On 7 Aug 2018, at 15:17, Deng Xiaodong <xd...@gmail.com> wrote:
>
> Hi Ben,
>
> If you would like to set multiple connections with the same *conn_id*, you
> can only do that in Web UI. That is, you need to build multiple entries
> with different host, but with the same *conn_id*.
>
> The method of setting connection in environment variables can only help set
> one connection for each *conn_id*.
>
> (Of course Web UI stores your connections in the metadata database. So if
> you really need to avoid manual interventions and need multiple connections
> for one single conn_id, hacking into the metadata database may be an
> option? even thought not recommended)
>
> Other guys please correct me if I'm wrong in any parts. Thanks.
>
>
> Regards,
> XD
>
> On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br...@gmail.com> wrote:
>
>> Hello -
>>
>> I'd like to define multiple connections for the `webhdfs_default` conn_id,
>> as we have a primary and backup host for the hadoop namenode. This is for
>> use with the WebHDFSHook, which queries for connections and seemingly can
>> accept a list of them from the database.
>>
>> ```
>> nn_connections = self.get_connections(self.webhdfs_conn_id)
>> for nn in nn_connections:
>> ```
>>
>> https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55
>>
>> I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
>> each hostname, however with the CLI and adding the entries to the
>> airflow.cfg I get errors that a connection already exists.
>>
>> It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
>> Is this a bug? Or is there a different way to accomplish without manually
>> adding via the UI?
>>
>> Thanks!
>> Ben Laird
>>
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.naik@reply.com
www.reply.com
Re: Multiple hosts for a single connection
Posted by Ash Berlin-Taylor <as...@apache.org>.
Hmm yes, it appears that the `airflow connections` CLI doesn't let you create multiple connections of the same conn_id. What the WebUI can do the CLI should be able to do also! It should allow that in some way (behind a `--allow-multiple` flag perhaps? I can see an argument for not allowing set multiple by default as it is often not what people want.)
-ash
> On 7 Aug 2018, at 15:17, Deng Xiaodong <xd...@gmail.com> wrote:
>
> Hi Ben,
>
> If you would like to set multiple connections with the same *conn_id*, you
> can only do that in Web UI. That is, you need to build multiple entries
> with different host, but with the same *conn_id*.
>
> The method of setting connection in environment variables can only help set
> one connection for each *conn_id*.
>
> (Of course Web UI stores your connections in the metadata database. So if
> you really need to avoid manual interventions and need multiple connections
> for one single conn_id, hacking into the metadata database may be an
> option? even thought not recommended)
>
> Other guys please correct me if I'm wrong in any parts. Thanks.
>
>
> Regards,
> XD
>
> On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br...@gmail.com> wrote:
>
>> Hello -
>>
>> I'd like to define multiple connections for the `webhdfs_default` conn_id,
>> as we have a primary and backup host for the hadoop namenode. This is for
>> use with the WebHDFSHook, which queries for connections and seemingly can
>> accept a list of them from the database.
>>
>> ```
>> nn_connections = self.get_connections(self.webhdfs_conn_id)
>> for nn in nn_connections:
>> ```
>>
>> https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55
>>
>> I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
>> each hostname, however with the CLI and adding the entries to the
>> airflow.cfg I get errors that a connection already exists.
>>
>> It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
>> Is this a bug? Or is there a different way to accomplish without manually
>> adding via the UI?
>>
>> Thanks!
>> Ben Laird
>>
Re: Multiple hosts for a single connection
Posted by Deng Xiaodong <xd...@gmail.com>.
Hi Ben,
If you would like to set multiple connections with the same *conn_id*, you
can only do that in Web UI. That is, you need to build multiple entries
with different host, but with the same *conn_id*.
The method of setting connection in environment variables can only help set
one connection for each *conn_id*.
(Of course Web UI stores your connections in the metadata database. So if
you really need to avoid manual interventions and need multiple connections
for one single conn_id, hacking into the metadata database may be an
option? even thought not recommended)
Other guys please correct me if I'm wrong in any parts. Thanks.
Regards,
XD
On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br...@gmail.com> wrote:
> Hello -
>
> I'd like to define multiple connections for the `webhdfs_default` conn_id,
> as we have a primary and backup host for the hadoop namenode. This is for
> use with the WebHDFSHook, which queries for connections and seemingly can
> accept a list of them from the database.
>
> ```
> nn_connections = self.get_connections(self.webhdfs_conn_id)
> for nn in nn_connections:
> ```
>
> https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55
>
> I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
> each hostname, however with the CLI and adding the entries to the
> airflow.cfg I get errors that a connection already exists.
>
> It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
> Is this a bug? Or is there a different way to accomplish without manually
> adding via the UI?
>
> Thanks!
> Ben Laird
>