You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/29 12:24:11 UTC
[GitHub] [airflow] potiuk opened a new issue #12702: Review connection/hook usage
potiuk opened a new issue #12702:
URL: https://github.com/apache/airflow/issues/12702
The discussion in https://github.com/apache/airflow/pull/12466#issuecomment-735347672 had shown that there is likely an expectation about making sure all the connections are 'registrable' in the UI of Airflow. This is not the case currently. Not all connections are visible in the UI (and possibly for a good reason) and currently connections are defined by Hook class variables.
Only a subset of the existing hooks provides this information, first of all because many of the connections/hooks do not require any custom fields, behaviour, they do not use automated Hook creation by connection type, but also it is because many of the hooks share the same connection type (For example almost all Google Hooks, most Amazon hooks etc).
Seems that we have `1 <> Many` relationships Connection/Hook rather than 1-1 as initially designed. Also, there is an existing "standard" introduced by the DBApi Hook where this metadata is stored in the Hook class as class-level field. Which we carried to those hooks that required to register themselves in the UI:
For example:
```
class ImapHook(BaseHook):
conn_name_attr = 'imap_conn_id'
default_conn_name = 'imap_default'
conn_type = 'imap'
conn_name = 'IMAP'
```
We likely need to address that by defining clear rules for the hooks/providers:
- whether we want to be able to map all hooks to connections
- whether we still want to create hook class automatically based on connection type - especially where we have multiple hooks sharing one connection type
- should we maybe have connection aliases to alias same connection type for different hooks
- where to keep that information (in Hooks ? Elsewehere)
- how to name the connection types - especially when there are siblings and multiple hooks sharing connection type.
I think those questions need to be answered before we can think about starting any implementation.
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
<!-- A short description of your feature -->
**Use case / motivation**
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
**Related Issues**
<!-- Is there currently another issue associated with this? -->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12702: Review connection/hook usage
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12702:
URL: https://github.com/apache/airflow/issues/12702#issuecomment-919073528
I think this is done now, right @potiuk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12702: Review connection/hook usage
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12702:
URL: https://github.com/apache/airflow/issues/12702#issuecomment-919185076
I think the problems about some inconsistent connection have been handled already. Part of them in separate provider changes and final step was done with Provider's managers refactoring and import optimisations. I think all the cases with 1<>Many relations which I was aware of have been handled - without setting clear rules though.
I think it is handled in a "good-enough" way for now. Any rule setting and enforcing them should be done when we migrate-away from WTforms and declarative way of defining connections in the UI with the new modern UI (or abandon the idea of configurable connections via UI, but that does not seem likely).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #12702: Review connection/hook usage
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #12702:
URL: https://github.com/apache/airflow/issues/12702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12702: Review connection/hook usage
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12702:
URL: https://github.com/apache/airflow/issues/12702#issuecomment-735385713
cc: @mik-laj @ashb -> if you want to tackle this, I described the situation and open questions we have for the Hook/Connections story. Marked it provisionally for 2.1. Feel free to take a lead on it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org