[jira] [Created] (AIRFLOW-7025) Fix SparkSqlHook.run_query to handle its parameter properly

["Kengo Seki (Jira)" <ji...@apache.org>]

Kengo Seki created AIRFLOW-7025:
-----------------------------------

             Summary: Fix SparkSqlHook.run_query to handle its parameter properly
                 Key: AIRFLOW-7025
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-7025
             Project: Apache Airflow
          Issue Type: Bug
          Components: hooks
    Affects Versions: 1.10.9
            Reporter: Kengo Seki
            Assignee: Kengo Seki


{{SparkSqlHook.run_query()}} has a parameter called {{cmd}}, which type is documented as string, but it doesn't work as expected.

{code}
In [1]: from airflow.providers.apache.spark.hooks.spark_sql import SparkSqlHook                                                                                              

In [2]: SparkSqlHook(sql="SELECT 1", master="local[*]", conn_id="spark_default").run_query(cmd="--help")                                                                     

(snip)

[2020-03-09 23:37:35,088] {spark_sql.py:149} INFO - b'  childArgs               [-e SELECT 1 - - h e l p]\n'
{code}

The passed argument "--help" is splitted into single characters. This is because the {{cmd}} parameter is concatenated to {{connection_cmd}} as a list of characters, as follows:

{code:title=airflow/providers/apache/spark/hooks/spark_sql.py}
 90     def _prepare_command(self, cmd):
 91         """
 92         Construct the spark-sql command to execute. Verbose output is enabled
 93         as default.
 94 
 95         :param cmd: command to append to the spark-sql command
 96         :type cmd: str
 97         :return: full command to be executed
 98         """
 99         connection_cmd = ["spark-sql"]

(snip)

130         connection_cmd += cmd
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)