[GitHub] [spark] hvanhovell commented on a diff in pull request #40762: [SPARK-42953][Connect][Followup] Fix maven test build for Scala client UDF tests

["hvanhovell (via GitHub)" <gi...@apache.org>]

hvanhovell commented on code in PR #40762:
URL: https://github.com/apache/spark/pull/40762#discussion_r1178100640


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/util/RemoteSparkSession.scala:
##########
@@ -102,6 +85,56 @@ object SparkConnectServerUtils {
     process
   }
 
+  /**
+   * As one shared spark will be started for all E2E tests, for tests that needs some special
+   * configs, we add them here
+   */
+  private def testConfigs: Seq[String] = {
+    // Use InMemoryTableCatalog for V2 writer tests
+    val writerV2Configs = {
+      val catalystTestJar = findJar( // To find InMemoryTableCatalog for V2 writer tests
+        "sql/catalyst",
+        "spark-catalyst",
+        "spark-catalyst",
+        test = true).getCanonicalPath
+      Seq(
+        "--jars",
+        catalystTestJar,
+        "--conf",
+        "spark.sql.catalog.testcat=org.apache.spark.sql.connector.catalog.InMemoryTableCatalog")
+    }
+
+    // Run tests using hive
+    val hiveTestConfigs = {
+      val catalogImplementation = if (IntegrationTestUtils.isSparkHiveJarAvailable) {
+        "hive"
+      } else {
+        // scalastyle:off println
+        println(
+          "Will start Spark Connect server with `spark.sql.catalogImplementation=in-memory`, " +
+            "some tests that rely on Hive will be ignored. If you don't want to skip them:\n" +
+            "1. Test with maven: run `build/mvn install -DskipTests -Phive` before testing\n" +
+            "2. Test with sbt: run test with `-Phive` profile")
+        // scalastyle:on println
+        "in-memory"
+      }
+      Seq("--conf", s"spark.sql.catalogImplementation=$catalogImplementation")
+    }
+
+    // For UDF maven E2E tests, the server needs the client code to find the UDFs defined in tests.
+    val udfTestConfigs = tryFindJar(
+      "connector/connect/client/jvm",
+      // SBT passes the client & test jars to the server process automatically.
+      // So we skip building or finding this jar for SBT.
+      "sbt-tests-do-not-need-this-jar",
+      "spark-connect-client-jvm",
+      test = true)
+      .map(clientTestJar => Seq("--jars", clientTestJar.getCanonicalPath))

Review Comment:
   The alternative is that we use the artifact upload functionality.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org