[DISCUSS] Connection extra field widgets: long vs short name convention

[Daniel Standish <dp...@gmail.com>]

For some connection types, UI customizations have been added so that you
have forms for the extra components.

E.g. with GCP we have extra__google_cloud_platform__project etc.

There's a PR for salesforce hook to make it so you that, when using secrets
backend you could use either the short or long name.  E.g. in the example
above it could be `extra__google_cloud_platform__project` or `project`.

GCP does not work like this; you must use long name.

I have always assumed that the convention is to do as is done in GCP: long
name only.

But I did a search and found a few others where we will take long or short
(wasb, azure data factory, snowflake).  But I'm not sure if this is just
for backward compat until next major release, or if it's meant to stay for
the long haul.

So the question is, what should be the convention?  Should we accept either
long or short, or stick with "there is preferably only one way to do a
given thing"?

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Jarek Potiuk <ja...@potiuk.com>]

> I don’t think what what being proposed by Daniel was to include both short and long names in the UI fields, but just in the get_conn method, right?

Hmm. That **might** work indeed. It will not be too helpful in having
both forms possible and making long form "default" for UI, "short"
default for other methods. It might add more confusion for the users -
because right now there is "one" blessed way for each connection (even
if the other way works, it is deprecated). Now we are trading it with
no "different in different connections" variance, with "where
connection is defined" variance.

No strong opinions on that one - it am fine with that if others see no
problem with this approach. It does not solve the root cause though so
it is another band-aid over band-aid though :).

J.

On Tue, Nov 23, 2021 at 12:12 AM Alex Begg <al...@gmail.com> wrote:
>
> Hi. I was actually the person who started this discussion on this PR: https://github.com/apache/airflow/pull/19530
>
> I understand the reasoning for the long names now, and I wanted to add to this discussion that if you make a hook's get_conn method allow for both short and long names, there isn't a necessity to alter the connection's extra names for the UI, right? So if the connection needs the long extra name for the UI customizations, you can replace every instance in the get_conn method where it access that extra name (e.g. extra__google__hello) with `extra.get('hello') or extra.get('extra__google__hello')` , and the UI customizations will still work and the people managing connections via secrets backends can use the shorter extra name.
>
> I don’t think what what being proposed by Daniel was to include both short and long names in the UI fields, but just in the get_conn method, right?
>
> The only time I find the long extra names not desirable is when you are defining connection values via URI (even as simple as via environment variables) and you have to include the long names in the URI.
>
> In my opinion all of our connections should allow for short names if you are not using the UI, since you can’t edit the connections via UI anyways.
>
> Alex
>
>
> On Mon, Nov 22, 2021 at 8:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Yeah. We really need to clean it up - but I am afraid without fixing
>> the UI for managing the connections in a sane way, the status quo
>> (even if it's not very clear how it works) will remain.
>>
>> J.
>>
>> On Mon, Nov 22, 2021 at 6:00 AM Josh Fell
>> <jo...@astronomer.io.invalid> wrote:
>> >
>> > To add a little wrinkle and small clarification here, there are some connection types that expose and use both classic Extra as well as customized fields in the connection form (Snowflake being the main example I can think of). As of 2.2, these two Extra options are being merged but not necessarily as a "smart" merge. If both are provided, the Extra JSON simply combines the classic Extra and custom field values.
>> >
>> > On Sun, Nov 21, 2021 at 1:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >>
>> >> Also - why the MUST:
>> >>
>> >> If you use custom fields, in the current implementation, when you have
>> >> both short and long form "available", when you edit such a connection,
>> >> the "extras" dictionary is overridden with long names (simply because
>> >> the JSON extra field is disabled and there is no "merge" logic for
>> >> handling both.
>> >>
>> >> So there is a scenario where you have a "short" name defined in extra
>> >> initially - but when you edit the connection, it will disappear
>> >> (unless you manually fill the "form" field with the same value).
>> >>
>> >> That's why for connections that implement "custom" field behaviours
>> >> the "short" form is really "deprecated" value (to support a case where
>> >> the user uses connection_uri or has a database configured with "short"
>> >> names from earlier manual "editing" the JSON field).
>> >>
>> >> And yes - it's not ideal, but I think the only proper way to handle it
>> >> is to implement better behaviour of the UI and support for
>> >> "namespaced" per-connection short names (which should not be too
>> >> difficult for anyone with some flair for javascript + Flask App
>> >> Builder).
>> >> Until then, we have to live with the dichotomy, I am afraid.
>> >>
>> >> J.
>> >>
>> >> On Sun, Nov 21, 2021 at 7:18 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >> >
>> >> > The problem is that if we use short extras, until we fix that in the
>> >> > UI, customizations won't work or they might override each other for
>> >> > different connections.
>> >> >
>> >> > So my take on it is - (until we fix the UI):
>> >> >
>> >> > a) if the connection customizes UI fields (so that you DO NOT edit the
>> >> > extras as JSON in the UI "extras" field manually) - you MUST use the
>> >> > long form.
>> >> > b) If the connection does not customize UI fields (so that you have to
>> >> > edit extras as JSIN in the UI "extras" field manually) - it is
>> >> > preferred to use short form
>> >> >
>> >> > J.
>> >> >
>> >> > On Sun, Nov 21, 2021 at 7:09 PM Daniel Standish <dp...@gmail.com> wrote:
>> >> > >
>> >> > > I wasn't exactly proposing updating the front end to handle short names.  It sounds like a reasonable idea but I am not trying to take that on right now.
>> >> > >
>> >> > > What I was trying to ask was, given the present form behavior, what should the hook behavior be, by convention?
>> >> > >
>> >> > > Specifically, should we write our conn parsing in hooks such that whenever there is a form extra (e.g. extra__google__hello) we always do `extra.get('hello') or extra.get('extra__google__hello')`
>> >> > >
>> >> > > Or something along those lines.
>> >> > >
>> >> > > Or should we stick with the status quo, which is to say there is no such convention and some hooks will accept both and others will not.
>> >> > >
>> >> > > I think my vote would be that by convention we should check both and short beats long in precedence.  This doesn't mean we have to go update every hook with form customizations.  But if we're reviewing a PR that checks both then it's ok to approve, and if we're reviewing a PR that only checks  long name we can suggest checking short name also.
>> >> > >
>> >> > > WDYT?
>> >> > >
>> >> > >

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Alex Begg <al...@gmail.com>]

Hi. I was actually the person who started this discussion on this PR:
https://github.com/apache/airflow/pull/19530

I understand the reasoning for the long names now, and I wanted to add to
this discussion that if you make a hook's get_conn method allow for both
short and long names, there isn't a necessity to alter the connection's
extra names for the UI, right? So if the connection needs the long extra
name for the UI customizations, you can replace every instance in the
get_conn method where it access that extra name (e.g. extra__google__hello)
with `extra.get('hello') or extra.get('extra__google__hello')` , and the UI
customizations will still work and the people managing connections via
secrets backends can use the shorter extra name.

I don’t think what what being proposed by Daniel was to include both short
and long names in the UI fields, but just in the get_conn method, right?

The only time I find the long extra names not desirable is when you are
defining connection values via URI (even as simple as via environment
variables) and you have to include the long names in the URI.

In my opinion all of our connections should allow for short names if you
are not using the UI, since you can’t edit the connections via UI anyways.

Alex


On Mon, Nov 22, 2021 at 8:01 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Yeah. We really need to clean it up - but I am afraid without fixing
> the UI for managing the connections in a sane way, the status quo
> (even if it's not very clear how it works) will remain.
>
> J.
>
> On Mon, Nov 22, 2021 at 6:00 AM Josh Fell
> <jo...@astronomer.io.invalid> wrote:
> >
> > To add a little wrinkle and small clarification here, there are some
> connection types that expose and use both classic Extra as well as
> customized fields in the connection form (Snowflake being the main example
> I can think of). As of 2.2, these two Extra options are being merged but
> not necessarily as a "smart" merge. If both are provided, the Extra JSON
> simply combines the classic Extra and custom field values.
> >
> > On Sun, Nov 21, 2021 at 1:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> Also - why the MUST:
> >>
> >> If you use custom fields, in the current implementation, when you have
> >> both short and long form "available", when you edit such a connection,
> >> the "extras" dictionary is overridden with long names (simply because
> >> the JSON extra field is disabled and there is no "merge" logic for
> >> handling both.
> >>
> >> So there is a scenario where you have a "short" name defined in extra
> >> initially - but when you edit the connection, it will disappear
> >> (unless you manually fill the "form" field with the same value).
> >>
> >> That's why for connections that implement "custom" field behaviours
> >> the "short" form is really "deprecated" value (to support a case where
> >> the user uses connection_uri or has a database configured with "short"
> >> names from earlier manual "editing" the JSON field).
> >>
> >> And yes - it's not ideal, but I think the only proper way to handle it
> >> is to implement better behaviour of the UI and support for
> >> "namespaced" per-connection short names (which should not be too
> >> difficult for anyone with some flair for javascript + Flask App
> >> Builder).
> >> Until then, we have to live with the dichotomy, I am afraid.
> >>
> >> J.
> >>
> >> On Sun, Nov 21, 2021 at 7:18 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >> >
> >> > The problem is that if we use short extras, until we fix that in the
> >> > UI, customizations won't work or they might override each other for
> >> > different connections.
> >> >
> >> > So my take on it is - (until we fix the UI):
> >> >
> >> > a) if the connection customizes UI fields (so that you DO NOT edit the
> >> > extras as JSON in the UI "extras" field manually) - you MUST use the
> >> > long form.
> >> > b) If the connection does not customize UI fields (so that you have to
> >> > edit extras as JSIN in the UI "extras" field manually) - it is
> >> > preferred to use short form
> >> >
> >> > J.
> >> >
> >> > On Sun, Nov 21, 2021 at 7:09 PM Daniel Standish <dp...@gmail.com>
> wrote:
> >> > >
> >> > > I wasn't exactly proposing updating the front end to handle short
> names.  It sounds like a reasonable idea but I am not trying to take that
> on right now.
> >> > >
> >> > > What I was trying to ask was, given the present form behavior, what
> should the hook behavior be, by convention?
> >> > >
> >> > > Specifically, should we write our conn parsing in hooks such that
> whenever there is a form extra (e.g. extra__google__hello) we always do
> `extra.get('hello') or extra.get('extra__google__hello')`
> >> > >
> >> > > Or something along those lines.
> >> > >
> >> > > Or should we stick with the status quo, which is to say there is no
> such convention and some hooks will accept both and others will not.
> >> > >
> >> > > I think my vote would be that by convention we should check both
> and short beats long in precedence.  This doesn't mean we have to go update
> every hook with form customizations.  But if we're reviewing a PR that
> checks both then it's ok to approve, and if we're reviewing a PR that only
> checks  long name we can suggest checking short name also.
> >> > >
> >> > > WDYT?
> >> > >
> >> > >
>

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Jarek Potiuk <ja...@potiuk.com>]

Yeah. We really need to clean it up - but I am afraid without fixing
the UI for managing the connections in a sane way, the status quo
(even if it's not very clear how it works) will remain.

J.

On Mon, Nov 22, 2021 at 6:00 AM Josh Fell
<jo...@astronomer.io.invalid> wrote:
>
> To add a little wrinkle and small clarification here, there are some connection types that expose and use both classic Extra as well as customized fields in the connection form (Snowflake being the main example I can think of). As of 2.2, these two Extra options are being merged but not necessarily as a "smart" merge. If both are provided, the Extra JSON simply combines the classic Extra and custom field values.
>
> On Sun, Nov 21, 2021 at 1:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> Also - why the MUST:
>>
>> If you use custom fields, in the current implementation, when you have
>> both short and long form "available", when you edit such a connection,
>> the "extras" dictionary is overridden with long names (simply because
>> the JSON extra field is disabled and there is no "merge" logic for
>> handling both.
>>
>> So there is a scenario where you have a "short" name defined in extra
>> initially - but when you edit the connection, it will disappear
>> (unless you manually fill the "form" field with the same value).
>>
>> That's why for connections that implement "custom" field behaviours
>> the "short" form is really "deprecated" value (to support a case where
>> the user uses connection_uri or has a database configured with "short"
>> names from earlier manual "editing" the JSON field).
>>
>> And yes - it's not ideal, but I think the only proper way to handle it
>> is to implement better behaviour of the UI and support for
>> "namespaced" per-connection short names (which should not be too
>> difficult for anyone with some flair for javascript + Flask App
>> Builder).
>> Until then, we have to live with the dichotomy, I am afraid.
>>
>> J.
>>
>> On Sun, Nov 21, 2021 at 7:18 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > The problem is that if we use short extras, until we fix that in the
>> > UI, customizations won't work or they might override each other for
>> > different connections.
>> >
>> > So my take on it is - (until we fix the UI):
>> >
>> > a) if the connection customizes UI fields (so that you DO NOT edit the
>> > extras as JSON in the UI "extras" field manually) - you MUST use the
>> > long form.
>> > b) If the connection does not customize UI fields (so that you have to
>> > edit extras as JSIN in the UI "extras" field manually) - it is
>> > preferred to use short form
>> >
>> > J.
>> >
>> > On Sun, Nov 21, 2021 at 7:09 PM Daniel Standish <dp...@gmail.com> wrote:
>> > >
>> > > I wasn't exactly proposing updating the front end to handle short names.  It sounds like a reasonable idea but I am not trying to take that on right now.
>> > >
>> > > What I was trying to ask was, given the present form behavior, what should the hook behavior be, by convention?
>> > >
>> > > Specifically, should we write our conn parsing in hooks such that whenever there is a form extra (e.g. extra__google__hello) we always do `extra.get('hello') or extra.get('extra__google__hello')`
>> > >
>> > > Or something along those lines.
>> > >
>> > > Or should we stick with the status quo, which is to say there is no such convention and some hooks will accept both and others will not.
>> > >
>> > > I think my vote would be that by convention we should check both and short beats long in precedence.  This doesn't mean we have to go update every hook with form customizations.  But if we're reviewing a PR that checks both then it's ok to approve, and if we're reviewing a PR that only checks  long name we can suggest checking short name also.
>> > >
>> > > WDYT?
>> > >
>> > >

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Josh Fell <jo...@astronomer.io.INVALID>]

To add a little wrinkle and small clarification here, there are some
connection types that expose and use both classic Extra as well as
customized fields in the connection form (Snowflake being the main example
I can think of). As of 2.2, these two Extra options are being merged but
not necessarily as a "smart" merge. If both are provided, the Extra JSON
simply combines the classic Extra and custom field values.

On Sun, Nov 21, 2021 at 1:30 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Also - why the MUST:
>
> If you use custom fields, in the current implementation, when you have
> both short and long form "available", when you edit such a connection,
> the "extras" dictionary is overridden with long names (simply because
> the JSON extra field is disabled and there is no "merge" logic for
> handling both.
>
> So there is a scenario where you have a "short" name defined in extra
> initially - but when you edit the connection, it will disappear
> (unless you manually fill the "form" field with the same value).
>
> That's why for connections that implement "custom" field behaviours
> the "short" form is really "deprecated" value (to support a case where
> the user uses connection_uri or has a database configured with "short"
> names from earlier manual "editing" the JSON field).
>
> And yes - it's not ideal, but I think the only proper way to handle it
> is to implement better behaviour of the UI and support for
> "namespaced" per-connection short names (which should not be too
> difficult for anyone with some flair for javascript + Flask App
> Builder).
> Until then, we have to live with the dichotomy, I am afraid.
>
> J.
>
> On Sun, Nov 21, 2021 at 7:18 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > The problem is that if we use short extras, until we fix that in the
> > UI, customizations won't work or they might override each other for
> > different connections.
> >
> > So my take on it is - (until we fix the UI):
> >
> > a) if the connection customizes UI fields (so that you DO NOT edit the
> > extras as JSON in the UI "extras" field manually) - you MUST use the
> > long form.
> > b) If the connection does not customize UI fields (so that you have to
> > edit extras as JSIN in the UI "extras" field manually) - it is
> > preferred to use short form
> >
> > J.
> >
> > On Sun, Nov 21, 2021 at 7:09 PM Daniel Standish <dp...@gmail.com>
> wrote:
> > >
> > > I wasn't exactly proposing updating the front end to handle short
> names.  It sounds like a reasonable idea but I am not trying to take that
> on right now.
> > >
> > > What I was trying to ask was, given the present form behavior, what
> should the hook behavior be, by convention?
> > >
> > > Specifically, should we write our conn parsing in hooks such that
> whenever there is a form extra (e.g. extra__google__hello) we always do
> `extra.get('hello') or extra.get('extra__google__hello')`
> > >
> > > Or something along those lines.
> > >
> > > Or should we stick with the status quo, which is to say there is no
> such convention and some hooks will accept both and others will not.
> > >
> > > I think my vote would be that by convention we should check both and
> short beats long in precedence.  This doesn't mean we have to go update
> every hook with form customizations.  But if we're reviewing a PR that
> checks both then it's ok to approve, and if we're reviewing a PR that only
> checks  long name we can suggest checking short name also.
> > >
> > > WDYT?
> > >
> > >
>

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Jarek Potiuk <ja...@potiuk.com>]

Also - why the MUST:

If you use custom fields, in the current implementation, when you have
both short and long form "available", when you edit such a connection,
the "extras" dictionary is overridden with long names (simply because
the JSON extra field is disabled and there is no "merge" logic for
handling both.

So there is a scenario where you have a "short" name defined in extra
initially - but when you edit the connection, it will disappear
(unless you manually fill the "form" field with the same value).

That's why for connections that implement "custom" field behaviours
the "short" form is really "deprecated" value (to support a case where
the user uses connection_uri or has a database configured with "short"
names from earlier manual "editing" the JSON field).

And yes - it's not ideal, but I think the only proper way to handle it
is to implement better behaviour of the UI and support for
"namespaced" per-connection short names (which should not be too
difficult for anyone with some flair for javascript + Flask App
Builder).
Until then, we have to live with the dichotomy, I am afraid.

J.

On Sun, Nov 21, 2021 at 7:18 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> The problem is that if we use short extras, until we fix that in the
> UI, customizations won't work or they might override each other for
> different connections.
>
> So my take on it is - (until we fix the UI):
>
> a) if the connection customizes UI fields (so that you DO NOT edit the
> extras as JSON in the UI "extras" field manually) - you MUST use the
> long form.
> b) If the connection does not customize UI fields (so that you have to
> edit extras as JSIN in the UI "extras" field manually) - it is
> preferred to use short form
>
> J.
>
> On Sun, Nov 21, 2021 at 7:09 PM Daniel Standish <dp...@gmail.com> wrote:
> >
> > I wasn't exactly proposing updating the front end to handle short names.  It sounds like a reasonable idea but I am not trying to take that on right now.
> >
> > What I was trying to ask was, given the present form behavior, what should the hook behavior be, by convention?
> >
> > Specifically, should we write our conn parsing in hooks such that whenever there is a form extra (e.g. extra__google__hello) we always do `extra.get('hello') or extra.get('extra__google__hello')`
> >
> > Or something along those lines.
> >
> > Or should we stick with the status quo, which is to say there is no such convention and some hooks will accept both and others will not.
> >
> > I think my vote would be that by convention we should check both and short beats long in precedence.  This doesn't mean we have to go update every hook with form customizations.  But if we're reviewing a PR that checks both then it's ok to approve, and if we're reviewing a PR that only checks  long name we can suggest checking short name also.
> >
> > WDYT?
> >
> >

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Jarek Potiuk <ja...@potiuk.com>]

The problem is that if we use short extras, until we fix that in the
UI, customizations won't work or they might override each other for
different connections.

So my take on it is - (until we fix the UI):

a) if the connection customizes UI fields (so that you DO NOT edit the
extras as JSON in the UI "extras" field manually) - you MUST use the
long form.
b) If the connection does not customize UI fields (so that you have to
edit extras as JSIN in the UI "extras" field manually) - it is
preferred to use short form

J.

On Sun, Nov 21, 2021 at 7:09 PM Daniel Standish <dp...@gmail.com> wrote:
>
> I wasn't exactly proposing updating the front end to handle short names.  It sounds like a reasonable idea but I am not trying to take that on right now.
>
> What I was trying to ask was, given the present form behavior, what should the hook behavior be, by convention?
>
> Specifically, should we write our conn parsing in hooks such that whenever there is a form extra (e.g. extra__google__hello) we always do `extra.get('hello') or extra.get('extra__google__hello')`
>
> Or something along those lines.
>
> Or should we stick with the status quo, which is to say there is no such convention and some hooks will accept both and others will not.
>
> I think my vote would be that by convention we should check both and short beats long in precedence.  This doesn't mean we have to go update every hook with form customizations.  But if we're reviewing a PR that checks both then it's ok to approve, and if we're reviewing a PR that only checks  long name we can suggest checking short name also.
>
> WDYT?
>
>

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Daniel Standish <dp...@gmail.com>]

I wasn't exactly proposing updating the front end to handle short names.
It sounds like a reasonable idea but I am not trying to take that on right
now.

What I was trying to ask was, given the present form behavior, what should
the *hook* behavior be, by convention?

Specifically, should we write our conn parsing in hooks such that whenever
there is a form extra (e.g. extra__google__hello) we always do
`extra.get('hello') or extra.get('extra__google__hello')`

Or something along those lines.

Or should we stick with the status quo, which is to say there is no such
convention and some hooks will accept both and others will not.

I think my vote would be that by convention we should check both and short
beats long in precedence.  This doesn't mean we have to go update every
hook with form customizations.  But if we're reviewing a PR that checks
both then it's ok to approve, and if we're reviewing a PR that only checks
long name we can suggest checking short name also.

WDYT?

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Ash Berlin-Taylor <as...@apache.org>]

Yeah, this has always been one of the thing that is a low-level 
annoyance/leaking abstraction to me, and if we can fix it in a 
backwards compatible way I'm all for it!

-ash

On Thu, Nov 11 2021 at 08:56:27 +0100, Jarek Potiuk <ja...@potiuk.com> 
wrote:
> As I understand and remember it, the "extra" prefix is needed because
> of the way the connection edit form works with extra fields.
> 
> The "extra__google_cloud_platform__" is a unique prefix used in the
> javascript to choose/validate the type of the extra parameter (custom
> form values).The way how the javascript works there is that all the
> types of fields from all the types of connections are retrieved from
> all "registered" connections and they are all added to the form, butt
> they are hidden by default. Only when the right "connection type" is
> selected from the form, all the matching "extra__" fields are shown
> (and hidden when another connection type is chosen) - thus providing
> the dynamic form. The extra_ prefix is there to add uniqueness to the
> id of the field and to match the field with the connection type.
> 
> The connections with the short names of extras simply do not have
> dynamic fields in the connection form. And those that you found
> supporting both, went from short to long in order to support the
> dynamic fields and the short ones are there to support backwards
> compatibility.
> 
> I'd be all for using the short name everywhere and converting the
> connection form to support it. But getting backwards compatibility
> might be a bit of a challenge - we have to make sure connections from
> old providers will work in the new airflow and the other way round.
> Also the old URLs with extra__ should work with the new solution as
> well. I believe it is possible, but requires some WTForms + javascript
> "magic" to make it works in a backwards-compatible way.
> 
> J.
> 
> 
> On Thu, Nov 11, 2021 at 5:31 AM Daniel Standish <dpstandish@gmail.com 
> <ma...@gmail.com>> wrote:
>> 
>>  For some connection types, UI customizations have been added so 
>> that you have forms for the extra components.
>> 
>>  E.g. with GCP we have extra__google_cloud_platform__project etc.
>> 
>>  There's a PR for salesforce hook to make it so you that, when using 
>> secrets backend you could use either the short or long name.  E.g. 
>> in the example above it could be 
>> `extra__google_cloud_platform__project` or `project`.
>> 
>>  GCP does not work like this; you must use long name.
>> 
>>  I have always assumed that the convention is to do as is done in 
>> GCP: long name only.
>> 
>>  But I did a search and found a few others where we will take long 
>> or short (wasb, azure data factory, snowflake).  But I'm not sure if 
>> this is just for backward compat until next major release, or if 
>> it's meant to stay for the long haul.
>> 
>>  So the question is, what should be the convention?  Should we 
>> accept either long or short, or stick with "there is preferably only 
>> one way to do a given thing"?

Re: [DISCUSS] Connection extra field widgets: long vs short name convention

[Jarek Potiuk <ja...@potiuk.com>]

As I understand and remember it, the "extra" prefix is needed because
of the way the connection edit form works with extra fields.

The "extra__google_cloud_platform__" is a unique prefix used in the
javascript to choose/validate the type of the extra parameter (custom
form values).The way how the javascript works there is that all the
types of fields from all the types of connections are retrieved from
all "registered" connections and they are all added to the form, butt
they are hidden by default. Only when the right "connection type" is
selected from the form, all the matching "extra__" fields are shown
(and hidden when another connection type is chosen) - thus providing
the dynamic form. The extra_ prefix is there to add uniqueness to the
id of the field and to match the field with the connection type.

The connections with the short names of extras simply do not have
dynamic fields in the connection form. And those that you found
supporting both, went from short to long in order to support the
dynamic fields and the short ones are there to support backwards
compatibility.

I'd be all for using the short name everywhere and converting the
connection form to support it. But getting backwards compatibility
might be a bit of a challenge - we have to make sure connections from
old providers will work in the new airflow and the other way round.
Also the old URLs with extra__ should work with the new solution as
well. I believe it is possible, but requires some WTForms + javascript
"magic" to make it works in a backwards-compatible way.

J.


On Thu, Nov 11, 2021 at 5:31 AM Daniel Standish <dp...@gmail.com> wrote:
>
> For some connection types, UI customizations have been added so that you have forms for the extra components.
>
> E.g. with GCP we have extra__google_cloud_platform__project etc.
>
> There's a PR for salesforce hook to make it so you that, when using secrets backend you could use either the short or long name.  E.g. in the example above it could be `extra__google_cloud_platform__project` or `project`.
>
> GCP does not work like this; you must use long name.
>
> I have always assumed that the convention is to do as is done in GCP: long name only.
>
> But I did a search and found a few others where we will take long or short (wasb, azure data factory, snowflake).  But I'm not sure if this is just for backward compat until next major release, or if it's meant to stay for the long haul.
>
> So the question is, what should be the convention?  Should we accept either long or short, or stick with "there is preferably only one way to do a given thing"?