[spark] branch branch-3.4 updated: [SPARK-42656][CONNECT][FOLLOWUP] Improve the script to start spark-connect server

[hv...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

hvanhovell pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.4 by this push:
     new d24be719dd6 [SPARK-42656][CONNECT][FOLLOWUP] Improve the script to start spark-connect server
d24be719dd6 is described below

commit d24be719dd6a4b5f3548cdb8cd7872cd7084dd82
Author: Zhen Li <zh...@users.noreply.github.com>
AuthorDate: Mon Mar 6 22:05:16 2023 -0400

    [SPARK-42656][CONNECT][FOLLOWUP] Improve the script to start spark-connect server
    
    ### What changes were proposed in this pull request?
    Make the server to start with Hive dependency and make the script to build with the correct scala version set in pom files.
    Also allow the server to take any extra spark configurations.
    
    ### Why are the changes needed?
    Adding hive dependency to allow some tests of write ops easier.
    Allow the server to start with the same Scala version set in the project.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Manually tested.
    
    Closes #40303 from zhenlineo/improve-scripts.
    
    Authored-by: Zhen Li <zh...@users.noreply.github.com>
    Signed-off-by: Herman van Hovell <he...@databricks.com>
    (cherry picked from commit d4818df19c73120edd9878907e87abaed20d4871)
    Signed-off-by: Herman van Hovell <he...@databricks.com>
---
 connector/connect/bin/spark-connect | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/connector/connect/bin/spark-connect b/connector/connect/bin/spark-connect
index 008209c8440..2f2ce7df08c 100755
--- a/connector/connect/bin/spark-connect
+++ b/connector/connect/bin/spark-connect
@@ -22,11 +22,12 @@ FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)"
 cd "$FWDIR"
 export SPARK_HOME=$FWDIR
 
-# Build the jars needed for spark submit and spark connect
-build/sbt package
-
 SCALA_BINARY_VER=`grep "scala.binary.version" "${SPARK_HOME}/pom.xml" | head -n1 | awk -F '[<>]' '{print $3}'`
+SCALA_ARG=$(if [ "${SCALA_BINARY_VER}" == "2.13" ]; then echo "-Pscala-2.13"; else echo ""; fi)
+
+# Build the jars needed for spark submit and spark connect
+build/sbt "${SCALA_ARG}" -Phive package
 
 CONNECT_JAR=`ls "${SPARK_HOME}"/connector/connect/server/target/scala-"${SCALA_BINARY_VER}"/spark-connect-assembly*.jar | paste -sd ',' -`
 
-exec "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR"
\ No newline at end of file
+exec "${SPARK_HOME}"/bin/spark-submit "$@" --class org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR"
\ No newline at end of file


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org