[GitHub] [airflow] ashb commented on a change in pull request #12710: Enable SparkSqlHook to use supplied connections

[GitBox <gi...@apache.org>]

ashb commented on a change in pull request #12710:
URL: https://github.com/apache/airflow/pull/12710#discussion_r551924009



##########
File path: airflow/providers/apache/spark/hooks/spark_sql.py
##########
@@ -44,15 +44,19 @@ class SparkSqlHook(BaseHook):
     :type executor_memory: str
     :param keytab: Full path to the file that contains the keytab
     :type keytab: str
-    :param master: spark://host:port, mesos://host:port, yarn, or local
+    :param master: (Deprecated) spark://host:port, mesos://host:port, yarn, or local
+        This parameter has been deprecated. Master and connection parameters (such as YARN queue)
+        are determined by the conn_id parameter.
     :type master: str
     :param name: Name of the job.
     :type name: str
     :param num_executors: Number of executors to launch
     :type num_executors: int
     :param verbose: Whether to pass the verbose flag to spark-sql
     :type verbose: bool
-    :param yarn_queue: The YARN queue to submit to (Default: "default")
+    :param yarn_queue: (Deprecated) The YARN queue to submit to (Default: "default")

Review comment:
       You haven't just deprecated this -- you've made it do nothing at all.
   
   Please make this argument still be respected if passed

##########
File path: airflow/providers/apache/spark/operators/spark_sql.py
##########
@@ -56,6 +58,8 @@ class SparkSqlOperator(BaseOperator):
     :param verbose: Whether to pass the verbose flag to spark-sql
     :type verbose: bool
     :param yarn_queue: The YARN queue to submit to (Default: "default")
+        This parameter has been deprecated. Master and connection parameters (such as YARN queue)

Review comment:
       I don't think we want to deprecate this --  it is a valid option to override the queue for just a single operator for instance.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org