Using the Sqoop Hook on Airflow

[Vijay Bhat <vi...@gmail.com>]

How do folks configure a Sqoop hook for the Sqoop operator? I don't see a
'sqoop' operator type in the list of connection types under Admin -> Create
-> Conn Type.

Do you just configure as an SSH hook and add all the fields under Extra? Or
is there a cleaner way?

Thanks!
Vijay

Re: Using the Sqoop Hook on Airflow

[Boris Tyukin <bo...@boristyukin.com>]

I just use bash operator and call sqoop cli. IMHO sqoop operator does not
add any functionality in fact it is limiting it:

1) you cannot use all sqoop commands / options
2) more importantly, I did not want to pass connection password in open
text to sqoop (if you do, it will be stored in log files)

So I just used bash operator *templatized *with jinja  and
hadoop.security.credential.provider with jceks to store passwords
http://www.ericlin.me/securely-managing-passwords-in-sqoop





On Mon, Feb 20, 2017 at 3:36 PM, Vijay Bhat <vi...@gmail.com> wrote:

> How do folks configure a Sqoop hook for the Sqoop operator? I don't see a
> 'sqoop' operator type in the list of connection types under Admin -> Create
> -> Conn Type.
>
> Do you just configure as an SSH hook and add all the fields under Extra? Or
> is there a cleaner way?
>
> Thanks!
> Vijay
>