[incubator-kyuubi] branch master updated: [KYUUBI #3600] [SPARK][K8S] Respect default static port for web UI on Spark K8s cluster mode

[ch...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git


The following commit(s) were added to refs/heads/master by this push:
     new e80a7db14 [KYUUBI #3600] [SPARK][K8S] Respect default static port for web UI on Spark K8s cluster mode
e80a7db14 is described below

commit e80a7db146095f383e2fce9b6fc16dfcb504323c
Author: zwangsheng <22...@qq.com>
AuthorDate: Mon Oct 10 19:59:08 2022 +0800

    [KYUUBI #3600] [SPARK][K8S] Respect default static port for web UI on Spark K8s cluster mode
    
    ### _Why are the changes needed?_
    
    Now kyuubi will help set web ui port to 0 while scalaing up spark sql engine, which is good for the general situation, help avoid port conflicts.
    
    But on k8s case, kyuubi should respect the user's default settings, here means web ui should be `4040`.
    
    ### _How was this patch tested?_
    - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
    
    - [ ] Add screenshots for manual tests if appropriate
    
    - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
    
    Closes #3600 from zwangsheng/improve/fix_spark_sql_engine_web_ui_on_k8s.
    
    Closes #3600
    
    0174e22f [zwangsheng] comment
    ac2dbedc [zwangsheng] fix
    1a2bff70 [zwangsheng] fix
    bd4c35c2 [zwangsheng] fix
    
    Authored-by: zwangsheng <22...@qq.com>
    Signed-off-by: Cheng Pan <ch...@apache.org>
---
 .../scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index 17b272315..3462c67ce 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -147,7 +147,6 @@ object SparkSQLEngine extends Logging {
     _sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold", "10000")
     _sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", "true")
     _sparkConf.setIfMissing("spark.master", "local")
-    _sparkConf.setIfMissing("spark.ui.port", "0")
     _sparkConf.set(
       "spark.redaction.regex",
       _sparkConf.get("spark.redaction.regex", "(?i)secret|password|token|access[.]key")
@@ -171,6 +170,11 @@ object SparkSQLEngine extends Logging {
       kyuubiConf.setIfMissing(FRONTEND_CONNECTION_URL_USE_HOSTNAME, false)
     }
 
+    // Set web ui port 0 to avoid port conflicts during non-k8s cluster mode
+    if (!isOnK8sClusterMode) {
+      _sparkConf.setIfMissing("spark.ui.port", "0")
+    }
+
     // Pass kyuubi config from spark with `spark.kyuubi`
     val sparkToKyuubiPrefix = "spark.kyuubi."
     _sparkConf.getAllWithPrefix(sparkToKyuubiPrefix).foreach { case (k, v) =>
@@ -309,4 +313,9 @@ object SparkSQLEngine extends Logging {
       },
       "CreateSparkTimeoutChecker").start()
   }
+
+  private def isOnK8sClusterMode: Boolean = {
+    // only spark driver pod will build with `SPARK_APPLICATION_ID` env.
+    Utils.isOnK8s && sys.env.contains("SPARK_APPLICATION_ID")
+  }
 }