[jira] [Commented] (AIRFLOW-985) Extend the sqoop operator/hook with additional parameters

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946231#comment-15946231 ] 

ASF subversion and git services commented on AIRFLOW-985:
---------------------------------------------------------

Commit 82eb20e9f525c09b7d8b4eea896dedcfb6b04f28 in incubator-airflow's branch refs/heads/master from [~Fokko]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=82eb20e ]

[AIRFLOW-985] Extend the sqoop operator and hook

The sqoop operator was a bit outdated and needed
some rework
including tests. Many lines have changed because
the code needed
some restructuring for better testing. Removed the
hive_home and
job_tracker because they are not used in any way
inside of the
sqoop class. Moved the num-mappers argument to the
constructor
because it is used for both importing and
exporting. Added
support for parquet. Added the ability to set the
driver and direct
mode and ability to pass jvm parameters to sqoop.

Closes #2177 from Fokko/airflow-985-extend-sqoop-
operator-hook


> Extend the sqoop operator/hook with additional parameters
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-985
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-985
>             Project: Apache Airflow
>          Issue Type: Bug
>            Reporter: Fokko Driesprong
>             Fix For: 1.9.0
>
>
> The current implementation of the sqoop hook/operator is rather inelaborate. For example, when exporting from hdfs to a rdbms, quite parameters are missing, e.g. it is not possible to set the format of the null values.
> Also some arguments can be extended, for example the current implementation does not support reading parquet.
> Beside all, tests need to be added to ensure proper behaviour.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)