[jira] [Work started] (AIRFLOW-770) HDFS hooks should support alternative ways of getting connection

["Daniel Huang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AIRFLOW-770 started by Daniel Huang.
--------------------------------------------
> HDFS hooks should support alternative ways of getting connection
> ----------------------------------------------------------------
>
>                 Key: AIRFLOW-770
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-770
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: hooks
>            Reporter: Daniel Huang
>            Assignee: Daniel Huang
>            Priority: Minor
>
> The HDFS hook currently uses {{get_connections()}} instead of {{get_connection()}} to grab the connection info. I believe this is so if multiple connections are specified, instead of choosing them at random, it appropriately passes them all via snakebite's HAClient.
> As far as I can tell, this means connection info can't be set outside of the UI, since environment variables are not looked at (which had me confused for a bit). I think ideally we'd want to be able to do so for the three different snakebite clients. Here are some possible suggestions for allowing this:
> * AutoConfigClient: add attribute like {{HDFSHook(..., autoconfig=True).get_conn()}}
> * Client: specify single URI in environment variable
> * HAClient: specify multiple URIs in environment variable, separated by commas? Not very adhering to standard and if we did this, we'd probably want to support this across all hooks.
> WebHDFS hook has a similar issue with pulling from env.
> references:
> https://github.com/apache/incubator-airflow/blob/b56cb5cc97de074bb0e520f66b79e7eb2d913fb1/airflow/hooks/base_hook.py#L43-L56
> https://github.com/apache/incubator-airflow/blob/b56cb5cc97de074bb0e520f66b79e7eb2d913fb1/airflow/hooks/hdfs_hook.py#L45-L73



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)