[jira] [Created] (AIRFLOW-2471) Fix HiveCliHook.load_df to use unused parameters

["Kengo Seki (JIRA)" <ji...@apache.org>]

Kengo Seki created AIRFLOW-2471:
-----------------------------------

             Summary: Fix HiveCliHook.load_df to use unused parameters
                 Key: AIRFLOW-2471
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2471
             Project: Apache Airflow
          Issue Type: Bug
          Components: hive_hooks, hooks
            Reporter: Kengo Seki
            Assignee: Kengo Seki


HiveCliHook.load_df has parameters called create and recreate:


{code}
    def load_df(
            self,
            df,
            table,
            create=True,
            recreate=False,

(snip)

        :param create: whether to create the table if it doesn't exist
        :type create: bool
        :param recreate: whether to drop and recreate the table at every
            execution
        :type recreate: bool
{code}

but these are already used as the default value.　For example, even if specifying {{recreate=True}}, {{DROP TABLE}} is not executed before {{CREATE TABLE}}.

{code}
In [1]: import pandas as pd

In [2]: from airflow.hooks.hive_hooks import HiveCliHook

In [3]: df = pd.DataFrame({"c": range(0, 10)})

In [4]: h = HiveCliHook()
[2018-05-16 10:27:55,814] {base_hook.py:83} INFO - Using connection to: localhost

In [5]: h.load_df(df, "t", recreate=True)
[2018-05-16 10:28:17,351] {hive_hooks.py:424} INFO - CREATE TABLE IF NOT EXISTS t (
c BIGINT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS textfile
;
[2018-05-16 10:28:17,353] {hive_hooks.py:217} INFO - beeline -u jdbc:hive2://localhost:10000/default;auth=none -f /tmp/airflow_hiveop__zc0kY/tmp99TFZK
[2018-05-16 10:28:19,730] {hive_hooks.py:232} INFO - Connecting to jdbc:hive2://localhost:10000/default;auth=none
[2018-05-16 10:28:20,127] {hive_hooks.py:232} INFO - Connected to: Apache Hive (version 1.2.1)
[2018-05-16 10:28:20,128] {hive_hooks.py:232} INFO - Driver: Hive JDBC (version 1.2.1)
[2018-05-16 10:28:20,129] {hive_hooks.py:232} INFO - Transaction isolation: TRANSACTION_REPEATABLE_READ
[2018-05-16 10:28:20,205] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> USE default;
[2018-05-16 10:28:20,446] {hive_hooks.py:232} INFO - No rows affected (0.234 seconds)
[2018-05-16 10:28:20,481] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> CREATE TABLE IF NOT EXISTS t (
[2018-05-16 10:28:20,485] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> c BIGINT)
[2018-05-16 10:28:20,491] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> ROW FORMAT DELIMITED
[2018-05-16 10:28:20,497] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> FIELDS TERMINATED BY ','
[2018-05-16 10:28:20,508] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> STORED AS textfile
[2018-05-16 10:28:20,582] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default> ;No rows affected (0.074 seconds)
[2018-05-16 10:28:20,597] {hive_hooks.py:232} INFO - 0: jdbc:hive2://localhost:10000/default>
[2018-05-16 10:28:20,598] {hive_hooks.py:232} INFO - Closing: 0: jdbc:hive2://localhost:10000/default;auth=none

(snip)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)