You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/03/19 23:53:00 UTC
[jira] [Commented] (AIRFLOW-3734) function `load_file` will not run
if param `partition` is None
[ https://issues.apache.org/jira/browse/AIRFLOW-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796628#comment-16796628 ]
ASF subversion and git services commented on AIRFLOW-3734:
----------------------------------------------------------
Commit e9a1bf8b6e85b19f38c6fd4e0e661e9b69cb7d70 in airflow's branch refs/heads/v1-10-stable from zhongjiajie
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=e9a1bf8 ]
[AIRFLOW-3734] Fix hql not run when partition is None (#4561)
> function `load_file` will not run if param `partition` is None
> --------------------------------------------------------------
>
> Key: AIRFLOW-3734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3734
> Project: Apache Airflow
> Issue Type: Bug
> Components: hive_hooks
> Affects Versions: 1.10.1
> Reporter: zhongjiajie
> Assignee: zhongjiajie
> Priority: Major
> Labels: easyfix, patch
> Fix For: 1.10.3
>
>
> the code in master branch in hive_hooks.py in function load_file like below
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals});"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += '\n'{code}
> when param partition is None, the hql will like this
> {code:java}
> LOAD DATA LOCAL INPATH <FILEPATH> OVERWRITE INTO TABLE <TABLE>\n{code}
> without `;` syntax the hql will not run, so I think we should move `;` from `hql += "PARTITION (\{pvals});"` to `hql += ';\n'`
>
> so the code will like
> {code:java}
> hql = "LOAD DATA LOCAL INPATH '{filepath}' "
> if overwrite:
> hql += "OVERWRITE "
> hql += "INTO TABLE {table} "
> if partition:
> pvals = ", ".join(
> ["{0}='{1}'".format(k, v) for k, v in partition.items()])
> hql += "PARTITION ({pvals})"
> # As a workaround for HIVE-10541, add a newline character
> # at the end of hql (AIRFLOW-2412).
> hql += ';\n'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)