[GitHub] [airflow] mikolaje commented on a change in pull request #6795: Adjust the MASTER_URL of spark-submit in SparkSubmitHook

[GitBox <gi...@apache.org>]

mikolaje commented on a change in pull request #6795: Adjust the MASTER_URL of spark-submit in SparkSubmitHook
URL: https://github.com/apache/airflow/pull/6795#discussion_r357001369
 
 

 ##########
 File path: airflow/contrib/hooks/spark_submit_hook.py
 ##########
 @@ -185,6 +185,8 @@ def _resolve_connection(self):
                 conn_data['master'] = "{}:{}".format(conn.host, conn.port)
             else:
                 conn_data['master'] = conn.host
+            if conn.uri:
+                conn_data['master'] = conn.uri
 
 Review comment:
   from my view, conn_data['master'] is equal to the MASTER_URL in `spark-submit` , the options are:
   ```
   spark://host:port, mesos://host:port, yarn, k8s://https://host:port, or local (Default: local[*]).
   ```
   `conn.uri`  probably comes from environment var `AIRFLOW_CONN_SPARK_DEFAULT`, which should be the same as one of options above, I guess.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services