You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (Jira)" <ji...@apache.org> on 2020/03/12 16:14:00 UTC
[jira] [Resolved] (AIRFLOW-7025) Fix SparkSqlHook.run_query to
handle its parameter properly
[ https://issues.apache.org/jira/browse/AIRFLOW-7025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kaxil Naik resolved AIRFLOW-7025.
---------------------------------
Fix Version/s: 1.10.10
Resolution: Fixed
> Fix SparkSqlHook.run_query to handle its parameter properly
> -----------------------------------------------------------
>
> Key: AIRFLOW-7025
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7025
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
> Affects Versions: 1.10.9
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Major
> Fix For: 1.10.10
>
>
> {{SparkSqlHook.run_query()}} has a parameter called {{cmd}}, which type is documented as string, but it doesn't work as expected.
> {code}
> In [1]: from airflow.providers.apache.spark.hooks.spark_sql import SparkSqlHook
> In [2]: SparkSqlHook(sql="SELECT 1", master="local[*]", conn_id="spark_default").run_query(cmd="--help")
> (snip)
> [2020-03-09 23:37:35,088] {spark_sql.py:149} INFO - b' childArgs [-e SELECT 1 - - h e l p]\n'
> {code}
> The passed argument "--help" is splitted into single characters. This is because the {{cmd}} parameter is concatenated to {{connection_cmd}} as a list of characters, as follows:
> {code:title=airflow/providers/apache/spark/hooks/spark_sql.py}
> 90 def _prepare_command(self, cmd):
> 91 """
> 92 Construct the spark-sql command to execute. Verbose output is enabled
> 93 as default.
> 94
> 95 :param cmd: command to append to the spark-sql command
> 96 :type cmd: str
> 97 :return: full command to be executed
> 98 """
> 99 connection_cmd = ["spark-sql"]
> (snip)
> 130 connection_cmd += cmd
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)