You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ash Berlin-Taylor <as...@apache.org> on 2020/04/24 10:31:34 UTC
[VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
As we discussed in the thread on Spec thread for our new API, the idea
came up of making conn_id unique in Airflow
<https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E>
The people in that thread seemed to be in favour of it:
- It is often confusing to users (there have been a few bug reports
about it over the years)
- It's questionable if it actually works well or not
- There are better/smarter tools for loadbalancing connections to a DB
than picking one of a random list
- For Hive at least it has been implemented another way - allowing two
host's in a single connection string
https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
- It makes the HTTP API confusing (needing a integer connection ID, and
a string conn_id field)
Given the downsides/work arounds, and the confusion this causes to I
propose we remove this (mis)feature from Airflow 2.0.
Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
in your vote/view, as I know AirBnB was using this in the past.
This vote will last for 96 Hours, until 10:30Z on 2020-04-27 https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
Every vote counts here, so please all vote.
Options:
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove with the reason
(or fractional values between -1..+1)
This is my +1 vote.
Thanks,
Ash
[RESULT] Make conn_id unique in Airflow (a.k.a. Remove
connection balancing HA )
Posted by Ash Berlin-Taylor <as...@apache.org>.
This vote has passed with 15 +1 votes, and zero -1 votes.
I've created https://github.com/apache/airflow/issues/8608
-ash
On Apr 24 2020, at 11:31 am, Ash Berlin-Taylor <as...@apache.org> wrote:
> As we discussed in the thread on Spec thread for our new API, the idea
> came up of making conn_id unique in Airflow
>
> <https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E>
>
> The people in that thread seemed to be in favour of it:
>
> - It is often confusing to users (there have been a few bug reports
> about it over the years)
> - It's questionable if it actually works well or not
> - There are better/smarter tools for loadbalancing connections to a DB
> than picking one of a random list
> - For Hive at least it has been implemented another way - allowing two
> host's in a single connection string
> https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> - It makes the HTTP API confusing (needing a integer connection ID, and
> a string conn_id field)
>
> Given the downsides/work arounds, and the confusion this causes to I
> propose we remove this (mis)feature from Airflow 2.0.
>
> Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> in your vote/view, as I know AirBnB was using this in the past.
>
> This vote will last for 96 Hours, until 10:30Z on 2020-04-27 https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
>
> Every vote counts here, so please all vote.
>
> Options:
>
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> (or fractional values between -1..+1)
>
> This is my +1 vote.
>
> Thanks,
> Ash
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection balancing HA )
Posted by Felix Uellendall <fe...@pm.me.INVALID>.
+1
Best regards,
Felix
Sent from ProtonMail Mobile
On Fri, Apr 24, 2020 at 17:11, Andrew Harmon <an...@gmail.com> wrote:
> +1 (non binding)
>
> Thanks,
> Andrew Harmon
>
> On Fri, Apr 24, 2020 at 10:24 AM Jiajie Zhong <zh...@hotmail.com>
> wrote:
>
>> +1
>>
>> Best Wish
>> — Jiajie
>>
>>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Andrew Harmon <an...@gmail.com>.
+1 (non binding)
Thanks,
Andrew Harmon
On Fri, Apr 24, 2020 at 10:24 AM Jiajie Zhong <zh...@hotmail.com>
wrote:
> +1
>
> Best Wish
> — Jiajie
>
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Jiajie Zhong <zh...@hotmail.com>.
+1
Best Wish
— Jiajie
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Kamil Breguła <ka...@polidea.com>.
+1
If a user really wants, then this feature can also be implemented
using a custom secret backend. This will allow for much better
behavior control. It doesn't have to be part of the core.
Best regards,
Kamil
On Fri, Apr 24, 2020 at 3:27 PM Bas Harenslak
<ba...@godatadriven.com.invalid> wrote:
>
> +1
>
> > On 24 Apr 2020, at 13:28, Jarek Potiuk <Ja...@polidea.com> wrote:
> >
> > +1
> >
> > On Fri, Apr 24, 2020 at 12:31 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> >> As we discussed in the thread on Spec thread for our new API, the idea
> >> came up of making conn_id unique in Airflow
> >>
> >> <
> >> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> >>>
> >>
> >> The people in that thread seemed to be in favour of it:
> >>
> >> - It is often confusing to users (there have been a few bug reports
> >> about it over the years)
> >> - It's questionable if it actually works well or not
> >> - There are better/smarter tools for loadbalancing connections to a DB
> >> than picking one of a random list
> >> - For Hive at least it has been implemented another way - allowing two
> >> host's in a single connection string
> >> https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> >> - It makes the HTTP API confusing (needing a integer connection ID, and
> >> a string conn_id field)
> >>
> >> Given the downsides/work arounds, and the confusion this causes to I
> >> propose we remove this (mis)feature from Airflow 2.0.
> >>
> >> Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> >> in your vote/view, as I know AirBnB was using this in the past.
> >>
> >> This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> >> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> >>
> >> Every vote counts here, so please all vote.
> >>
> >> Options:
> >>
> >>
> >> [ ] +1 approve
> >> [ ] +0 no opinion
> >> [ ] -1 disapprove with the reason
> >>
> >> (or fractional values between -1..+1)
> >>
> >> This is my +1 vote.
> >>
> >> Thanks,
> >> Ash
> >>
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Bas Harenslak <ba...@godatadriven.com.INVALID>.
+1
> On 24 Apr 2020, at 13:28, Jarek Potiuk <Ja...@polidea.com> wrote:
>
> +1
>
> On Fri, Apr 24, 2020 at 12:31 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> As we discussed in the thread on Spec thread for our new API, the idea
>> came up of making conn_id unique in Airflow
>>
>> <
>> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
>>>
>>
>> The people in that thread seemed to be in favour of it:
>>
>> - It is often confusing to users (there have been a few bug reports
>> about it over the years)
>> - It's questionable if it actually works well or not
>> - There are better/smarter tools for loadbalancing connections to a DB
>> than picking one of a random list
>> - For Hive at least it has been implemented another way - allowing two
>> host's in a single connection string
>> https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
>> - It makes the HTTP API confusing (needing a integer connection ID, and
>> a string conn_id field)
>>
>> Given the downsides/work arounds, and the confusion this causes to I
>> propose we remove this (mis)feature from Airflow 2.0.
>>
>> Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
>> in your vote/view, as I know AirBnB was using this in the past.
>>
>> This vote will last for 96 Hours, until 10:30Z on 2020-04-27
>> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
>>
>> Every vote counts here, so please all vote.
>>
>> Options:
>>
>>
>> [ ] +1 approve
>> [ ] +0 no opinion
>> [ ] -1 disapprove with the reason
>>
>> (or fractional values between -1..+1)
>>
>> This is my +1 vote.
>>
>> Thanks,
>> Ash
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Jarek Potiuk <Ja...@polidea.com>.
+1
On Fri, Apr 24, 2020 at 12:31 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> As we discussed in the thread on Spec thread for our new API, the idea
> came up of making conn_id unique in Airflow
>
> <
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> >
>
> The people in that thread seemed to be in favour of it:
>
> - It is often confusing to users (there have been a few bug reports
> about it over the years)
> - It's questionable if it actually works well or not
> - There are better/smarter tools for loadbalancing connections to a DB
> than picking one of a random list
> - For Hive at least it has been implemented another way - allowing two
> host's in a single connection string
> https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> - It makes the HTTP API confusing (needing a integer connection ID, and
> a string conn_id field)
>
> Given the downsides/work arounds, and the confusion this causes to I
> propose we remove this (mis)feature from Airflow 2.0.
>
> Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> in your vote/view, as I know AirBnB was using this in the past.
>
> This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
>
> Every vote counts here, so please all vote.
>
> Options:
>
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> (or fractional values between -1..+1)
>
> This is my +1 vote.
>
> Thanks,
> Ash
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
+1 (binding)
Thank you Ash!
Op vr 24 apr. 2020 om 12:41 schreef Robin Edwards <ro...@bidnamic.com>:
> +1
>
> On Fri, 24 Apr 2020, 11:32 Kaxil Naik, <ka...@gmail.com> wrote:
>
> > +1
> >
> > On Fri, Apr 24, 2020 at 11:31 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > As we discussed in the thread on Spec thread for our new API, the idea
> > > came up of making conn_id unique in Airflow
> > >
> > > <
> > >
> >
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> > > >
> > >
> > > The people in that thread seemed to be in favour of it:
> > >
> > > - It is often confusing to users (there have been a few bug reports
> > > about it over the years)
> > > - It's questionable if it actually works well or not
> > > - There are better/smarter tools for loadbalancing connections to a DB
> > > than picking one of a random list
> > > - For Hive at least it has been implemented another way - allowing two
> > > host's in a single connection string
> > > https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> > > - It makes the HTTP API confusing (needing a integer connection ID, and
> > > a string conn_id field)
> > >
> > > Given the downsides/work arounds, and the confusion this causes to I
> > > propose we remove this (mis)feature from Airflow 2.0.
> > >
> > > Kevin from AirBnb (on bcc, hopefully you see this) I would be
> interested
> > > in your vote/view, as I know AirBnB was using this in the past.
> > >
> > > This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> > >
> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> > >
> > > Every vote counts here, so please all vote.
> > >
> > > Options:
> > >
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove with the reason
> > >
> > > (or fractional values between -1..+1)
> > >
> > > This is my +1 vote.
> > >
> > > Thanks,
> > > Ash
> > >
> >
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Robin Edwards <ro...@bidnamic.com>.
+1
On Fri, 24 Apr 2020, 11:32 Kaxil Naik, <ka...@gmail.com> wrote:
> +1
>
> On Fri, Apr 24, 2020 at 11:31 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > As we discussed in the thread on Spec thread for our new API, the idea
> > came up of making conn_id unique in Airflow
> >
> > <
> >
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> > >
> >
> > The people in that thread seemed to be in favour of it:
> >
> > - It is often confusing to users (there have been a few bug reports
> > about it over the years)
> > - It's questionable if it actually works well or not
> > - There are better/smarter tools for loadbalancing connections to a DB
> > than picking one of a random list
> > - For Hive at least it has been implemented another way - allowing two
> > host's in a single connection string
> > https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> > - It makes the HTTP API confusing (needing a integer connection ID, and
> > a string conn_id field)
> >
> > Given the downsides/work arounds, and the confusion this causes to I
> > propose we remove this (mis)feature from Airflow 2.0.
> >
> > Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> > in your vote/view, as I know AirBnB was using this in the past.
> >
> > This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> > https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> >
> > Every vote counts here, so please all vote.
> >
> > Options:
> >
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> > (or fractional values between -1..+1)
> >
> > This is my +1 vote.
> >
> > Thanks,
> > Ash
> >
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Kaxil Naik <ka...@gmail.com>.
+1
On Fri, Apr 24, 2020 at 11:31 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> As we discussed in the thread on Spec thread for our new API, the idea
> came up of making conn_id unique in Airflow
>
> <
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> >
>
> The people in that thread seemed to be in favour of it:
>
> - It is often confusing to users (there have been a few bug reports
> about it over the years)
> - It's questionable if it actually works well or not
> - There are better/smarter tools for loadbalancing connections to a DB
> than picking one of a random list
> - For Hive at least it has been implemented another way - allowing two
> host's in a single connection string
> https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> - It makes the HTTP API confusing (needing a integer connection ID, and
> a string conn_id field)
>
> Given the downsides/work arounds, and the confusion this causes to I
> propose we remove this (mis)feature from Airflow 2.0.
>
> Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> in your vote/view, as I know AirBnB was using this in the past.
>
> This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
>
> Every vote counts here, so please all vote.
>
> Options:
>
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> (or fractional values between -1..+1)
>
> This is my +1 vote.
>
> Thanks,
> Ash
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Kevin Yang <yr...@gmail.com>.
+1 (binding)
Thank you Ash for asking specifically!🙏
We do use it as a hacky load balancing mechanism in Airbnb but I agree that
the right thing to is to make it unique, we will migrate to a proper
solution.
Cheers,
Kevin Y
On Fri, Apr 24, 2020 at 10:09 AM Daniel Standish <dp...@gmail.com>
wrote:
> +1 (non-binding) :)
>
>
> On Fri, Apr 24, 2020 at 10:00 AM Xinbin Huang <bi...@gmail.com>
> wrote:
>
> > +1 non-binding
> >
> > Best,
> > Bin
> >
> > On Fri, Apr 24, 2020 at 9:20 AM Tomasz Urbaszek <
> > tomasz.urbaszek@polidea.com>
> > wrote:
> >
> > > +1 binding
> > >
> > > Tomek
> > >
> > >
> > > On Fri, Apr 24, 2020 at 6:15 PM QP Hou <qp...@scribd.com> wrote:
> > > >
> > > > +1 (non binding)
> > > >
> > > > On Fri, Apr 24, 2020 at 3:31 AM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > > >
> > > > > As we discussed in the thread on Spec thread for our new API, the
> > idea
> > > > > came up of making conn_id unique in Airflow
> > > > >
> > > > > <
> > >
> >
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> > > >
> > > > >
> > > > > The people in that thread seemed to be in favour of it:
> > > > >
> > > > > - It is often confusing to users (there have been a few bug reports
> > > > > about it over the years)
> > > > > - It's questionable if it actually works well or not
> > > > > - There are better/smarter tools for loadbalancing connections to a
> > DB
> > > > > than picking one of a random list
> > > > > - For Hive at least it has been implemented another way - allowing
> > two
> > > > > host's in a single connection string
> > > > > https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> > > > > - It makes the HTTP API confusing (needing a integer connection ID,
> > and
> > > > > a string conn_id field)
> > > > >
> > > > > Given the downsides/work arounds, and the confusion this causes to
> I
> > > > > propose we remove this (mis)feature from Airflow 2.0.
> > > > >
> > > > > Kevin from AirBnb (on bcc, hopefully you see this) I would be
> > > interested
> > > > > in your vote/view, as I know AirBnB was using this in the past.
> > > > >
> > > > > This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> > >
> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> > > > >
> > > > > Every vote counts here, so please all vote.
> > > > >
> > > > > Options:
> > > > >
> > > > >
> > > > > [ ] +1 approve
> > > > > [ ] +0 no opinion
> > > > > [ ] -1 disapprove with the reason
> > > > >
> > > > > (or fractional values between -1..+1)
> > > > >
> > > > > This is my +1 vote.
> > > > >
> > > > > Thanks,
> > > > > Ash
> > >
> > >
> > >
> > > --
> > >
> > > Tomasz Urbaszek
> > > Polidea | Software Engineer
> > >
> > > M: +48 505 628 493
> > > E: tomasz.urbaszek@polidea.com
> > >
> > > Unique Tech
> > > Check out our projects!
> > >
> >
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Daniel Standish <dp...@gmail.com>.
+1 (non-binding) :)
On Fri, Apr 24, 2020 at 10:00 AM Xinbin Huang <bi...@gmail.com> wrote:
> +1 non-binding
>
> Best,
> Bin
>
> On Fri, Apr 24, 2020 at 9:20 AM Tomasz Urbaszek <
> tomasz.urbaszek@polidea.com>
> wrote:
>
> > +1 binding
> >
> > Tomek
> >
> >
> > On Fri, Apr 24, 2020 at 6:15 PM QP Hou <qp...@scribd.com> wrote:
> > >
> > > +1 (non binding)
> > >
> > > On Fri, Apr 24, 2020 at 3:31 AM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > > >
> > > > As we discussed in the thread on Spec thread for our new API, the
> idea
> > > > came up of making conn_id unique in Airflow
> > > >
> > > > <
> >
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> > >
> > > >
> > > > The people in that thread seemed to be in favour of it:
> > > >
> > > > - It is often confusing to users (there have been a few bug reports
> > > > about it over the years)
> > > > - It's questionable if it actually works well or not
> > > > - There are better/smarter tools for loadbalancing connections to a
> DB
> > > > than picking one of a random list
> > > > - For Hive at least it has been implemented another way - allowing
> two
> > > > host's in a single connection string
> > > > https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> > > > - It makes the HTTP API confusing (needing a integer connection ID,
> and
> > > > a string conn_id field)
> > > >
> > > > Given the downsides/work arounds, and the confusion this causes to I
> > > > propose we remove this (mis)feature from Airflow 2.0.
> > > >
> > > > Kevin from AirBnb (on bcc, hopefully you see this) I would be
> > interested
> > > > in your vote/view, as I know AirBnB was using this in the past.
> > > >
> > > > This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> > https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> > > >
> > > > Every vote counts here, so please all vote.
> > > >
> > > > Options:
> > > >
> > > >
> > > > [ ] +1 approve
> > > > [ ] +0 no opinion
> > > > [ ] -1 disapprove with the reason
> > > >
> > > > (or fractional values between -1..+1)
> > > >
> > > > This is my +1 vote.
> > > >
> > > > Thanks,
> > > > Ash
> >
> >
> >
> > --
> >
> > Tomasz Urbaszek
> > Polidea | Software Engineer
> >
> > M: +48 505 628 493
> > E: tomasz.urbaszek@polidea.com
> >
> > Unique Tech
> > Check out our projects!
> >
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Xinbin Huang <bi...@gmail.com>.
+1 non-binding
Best,
Bin
On Fri, Apr 24, 2020 at 9:20 AM Tomasz Urbaszek <to...@polidea.com>
wrote:
> +1 binding
>
> Tomek
>
>
> On Fri, Apr 24, 2020 at 6:15 PM QP Hou <qp...@scribd.com> wrote:
> >
> > +1 (non binding)
> >
> > On Fri, Apr 24, 2020 at 3:31 AM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> > >
> > > As we discussed in the thread on Spec thread for our new API, the idea
> > > came up of making conn_id unique in Airflow
> > >
> > > <
> https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E
> >
> > >
> > > The people in that thread seemed to be in favour of it:
> > >
> > > - It is often confusing to users (there have been a few bug reports
> > > about it over the years)
> > > - It's questionable if it actually works well or not
> > > - There are better/smarter tools for loadbalancing connections to a DB
> > > than picking one of a random list
> > > - For Hive at least it has been implemented another way - allowing two
> > > host's in a single connection string
> > > https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> > > - It makes the HTTP API confusing (needing a integer connection ID, and
> > > a string conn_id field)
> > >
> > > Given the downsides/work arounds, and the confusion this causes to I
> > > propose we remove this (mis)feature from Airflow 2.0.
> > >
> > > Kevin from AirBnb (on bcc, hopefully you see this) I would be
> interested
> > > in your vote/view, as I know AirBnB was using this in the past.
> > >
> > > This vote will last for 96 Hours, until 10:30Z on 2020-04-27
> https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> > >
> > > Every vote counts here, so please all vote.
> > >
> > > Options:
> > >
> > >
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove with the reason
> > >
> > > (or fractional values between -1..+1)
> > >
> > > This is my +1 vote.
> > >
> > > Thanks,
> > > Ash
>
>
>
> --
>
> Tomasz Urbaszek
> Polidea | Software Engineer
>
> M: +48 505 628 493
> E: tomasz.urbaszek@polidea.com
>
> Unique Tech
> Check out our projects!
>
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by Tomasz Urbaszek <to...@polidea.com>.
+1 binding
Tomek
On Fri, Apr 24, 2020 at 6:15 PM QP Hou <qp...@scribd.com> wrote:
>
> +1 (non binding)
>
> On Fri, Apr 24, 2020 at 3:31 AM Ash Berlin-Taylor <as...@apache.org> wrote:
> >
> > As we discussed in the thread on Spec thread for our new API, the idea
> > came up of making conn_id unique in Airflow
> >
> > <https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E>
> >
> > The people in that thread seemed to be in favour of it:
> >
> > - It is often confusing to users (there have been a few bug reports
> > about it over the years)
> > - It's questionable if it actually works well or not
> > - There are better/smarter tools for loadbalancing connections to a DB
> > than picking one of a random list
> > - For Hive at least it has been implemented another way - allowing two
> > host's in a single connection string
> > https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> > - It makes the HTTP API confusing (needing a integer connection ID, and
> > a string conn_id field)
> >
> > Given the downsides/work arounds, and the confusion this causes to I
> > propose we remove this (mis)feature from Airflow 2.0.
> >
> > Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> > in your vote/view, as I know AirBnB was using this in the past.
> >
> > This vote will last for 96 Hours, until 10:30Z on 2020-04-27 https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
> >
> > Every vote counts here, so please all vote.
> >
> > Options:
> >
> >
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove with the reason
> >
> > (or fractional values between -1..+1)
> >
> > This is my +1 vote.
> >
> > Thanks,
> > Ash
--
Tomasz Urbaszek
Polidea | Software Engineer
M: +48 505 628 493
E: tomasz.urbaszek@polidea.com
Unique Tech
Check out our projects!
Re: [VOTE] Make conn_id unique in Airflow (a.k.a. Remove connection
balancing HA )
Posted by QP Hou <qp...@scribd.com>.
+1 (non binding)
On Fri, Apr 24, 2020 at 3:31 AM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> As we discussed in the thread on Spec thread for our new API, the idea
> came up of making conn_id unique in Airflow
>
> <https://lists.apache.org/thread.html/rfb6f7d95b2754fda9dd09b08444214dfad12d10f143d32de0fcf4104%40%3Cdev.airflow.apache.org%3E>
>
> The people in that thread seemed to be in favour of it:
>
> - It is often confusing to users (there have been a few bug reports
> about it over the years)
> - It's questionable if it actually works well or not
> - There are better/smarter tools for loadbalancing connections to a DB
> than picking one of a random list
> - For Hive at least it has been implemented another way - allowing two
> host's in a single connection string
> https://github.com/apache/airflow/pull/4708 (available in 1.10.6)
> - It makes the HTTP API confusing (needing a integer connection ID, and
> a string conn_id field)
>
> Given the downsides/work arounds, and the confusion this causes to I
> propose we remove this (mis)feature from Airflow 2.0.
>
> Kevin from AirBnb (on bcc, hopefully you see this) I would be interested
> in your vote/view, as I know AirBnB was using this in the past.
>
> This vote will last for 96 Hours, until 10:30Z on 2020-04-27 https://www.timeanddate.com/worldclock/fixedtime.html?iso=20200427T1030
>
> Every vote counts here, so please all vote.
>
> Options:
>
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove with the reason
>
> (or fractional values between -1..+1)
>
> This is my +1 vote.
>
> Thanks,
> Ash