[jira] [Commented] (AIRFLOW-1770) Add option to file and hiveconfs in HiveOperator

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323775#comment-16323775 ] 

ASF subversion and git services commented on AIRFLOW-1770:
----------------------------------------------------------

Commit eb994d683f244f63dd191a6640baaee66ffc8e29 in incubator-airflow's branch refs/heads/master from [~alanma]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=eb994d6 ]

[AIRFLOW-1770] Allow HiveOperator to take in a file

Clarify and upgrade HiveOperator. Include
description of hql parameter being able to
take in a relative path from the dag file
of a hive script, templated or not. Add
ability to template hiveconf variables. Add
default value to the map reduce job name as
well as add updated hiveconf var for queue.

Closes #2752 from wolfier/AIRFLOW-1770


> Add option to file and hiveconfs in HiveOperator
> ------------------------------------------------
>
>                 Key: AIRFLOW-1770
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1770
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Ace Haidrey
>            Assignee: Alan Ma
>              Labels: operators
>             Fix For: 1.10.0, 1.9.1, 2.0.0
>
>
> The HiveOperator as it currently stands is not flexible enough to accept a hive file and operate on that. You need to read in the contents and pass it and if you do that you need to change the way hiveconfs are in your file to jinja templating.
> Many teams already have their existing sql/hql files and don't want to convert them to make them as portable and decoupled as possible.
> To accomplish this all we need to do is add the option to pass a hql_file and hiveconfs to the HiveOperator. We change the code in the execute to throw an error if both a hql_file and an hql statement are passed. If just hql_file the simplest way without changing the code of the hive hook is to just read the content of the hql_file and set it to be the self.hql. The hiveconfs get passed directly to the run_cli method and we can combine them with the already passed in hiveconfs.
> If we want to make it optional to pass in the context as hiveconfs we can add that too as related to AIRFLOW-788.
> I've included some simple tests to show it all works how we expect.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)