You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Vladislav Glinskiy (JIRA)" <ji...@apache.org> on 2018/09/12 09:24:00 UTC
[jira] [Created] (AIRFLOW-3047) HiveCliHook does not work properly
with Beeline
Vladislav Glinskiy created AIRFLOW-3047:
-------------------------------------------
Summary: HiveCliHook does not work properly with Beeline
Key: AIRFLOW-3047
URL: https://issues.apache.org/jira/browse/AIRFLOW-3047
Project: Apache Airflow
Issue Type: Bug
Components: hive_hooks, hooks
Affects Versions: 1.10.0
Reporter: Vladislav Glinskiy
Simple _HiveOperator_ does not work properly in the case when _hive_cli_default_ connection configured to use _Beeline_.
*Steps to reproduce:*
1. Setup Hive/HiveServer2 and Airflow environment with _beeline_ in _PATH_
2. Create test _datetimes_ table
As example:
{code:java}
CREATE EXTERNAL TABLE datetimes (
datetimes STRING)
STORED AS PARQUET
LOCATION '/opt/apps/datetimes';{code}
3. Edit _hive_cli_default_ connection:
{code:java}
airflow connections --delete --conn_id hive_cli_default
airflow connections --add --conn_id hive_cli_default --conn_type hive_cli --conn_host $HOST --conn_port 10000 --conn_schema default --conn_login $CONN_LOGIN --conn_password $CONN_PASSWORD --conn_extra "{\"use_beeline\": true, \"auth\": \"null;user=$HS_USER;password=$HS_PASSWORD\"}"
{code}
Set variables according to your environment.
4. Create simple DAG:
{code:java}
"""
###
Sample DAG, which declares single Hive task.
"""
import datetime
import airflow
from airflow import DAG
from airflow.operators.hive_operator import HiveOperator
from datetime import timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': airflow.utils.dates.days_ago(0, hour=0, minute=0, second=1),
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'provide_context': True
}
dag = DAG(
'hive_task_dag',
default_args=default_args,
description='Single task DAG',
schedule_interval=timedelta(minutes=15))
insert_current_datetime = HiveOperator(
task_id='insert_current_datetime_task',
hql="insert into table datetimes values ('" + datetime.datetime.now().strftime("%I:%M%p on %B %d, %Y") + "');",
dag=dag)
dag.doc_md = __doc__
{code}
5. Trigger DAG execution. Ensure that DAG completes successfully.
6. Check _datetimes_ table. It will be empty.
As it turned out the issue is caused by an invalid temporary script file. The problem will be fixed if we add new-line character at the end of the script.
So, a possible fix is to change:
*hive_hooks.py:182*
{code:java}
if schema:
hql = "USE {schema};\n{hql}".format(**locals())
{code}
to
{code:java}
if schema:
hql = "USE {schema};\n{hql}\n".format(**locals())
{code}
Don't know how it can affect _hive shell_ queries since it is tested only against _beeline_.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)