You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2026/04/04 06:01:14 UTC
[PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
sarutak opened a new pull request, #55192:
URL: https://github.com/apache/spark/pull/55192
### What changes were proposed in this pull request?
This PR is a part of SPARK-56352 for Java-based test JARs, replacing pre-built test JAR files containing Java classes with dynamic compilation at test time, removing 10 binary JAR files from the repository.
Changes:
- Add `TestUtils.createJarWithJavaSources()` in `SparkTestUtils.scala` that
compiles Java source code with `javac` and packages the resulting classes
into a JAR.
- Externalize Java source files for Hive UDFs/UDAFs/UDTFs to
`src/test/resources/` (9 files), loaded at test time via
`getContextClassLoader.getResource()`. These source files were
reverse-engineered from the class files in the pre-built JARs, as no
original source code existed in the repository.
- Update test suites to use dynamically generated JARs instead of pre-built
ones.
- Remove deleted JAR entries from `dev/test-jars.txt`.
JARs removed:
- `core/src/test/resources/TestUDTF.jar`
- `core/src/test/resources/TestHelloV2_2.13.jar`
- `core/src/test/resources/TestHelloV3_2.13.jar`
- `sql/connect/client/jvm/src/test/resources/TestHelloV2_2.13.jar`
- `sql/core/src/test/resources/SPARK-33084.jar`
- `sql/hive-thriftserver/src/test/resources/TestUDTF.jar`
- `sql/hive/src/test/noclasspath/hive-test-udfs.jar`
- `sql/hive/src/test/resources/SPARK-21101-1.0.jar`
- `sql/hive/src/test/resources/TestUDTF.jar`
- `sql/hive/src/test/resources/data/files/TestSerDe.jar`
### Why are the changes needed?
As noted in the PR discussion (https://github.com/apache/spark/pull/50378):
> the ultimate goal is to refactor the tests to automatically build the jars
> instead of using pre-built ones
This PR achieves that goal for all Java-based test JARs. By generating JARs dynamically at test time, no binary artifacts need to be stored in the source tree, and the release-time workaround becomes unnecessary for these files.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
All affected test suites pass:
- SparkContextSuite, SparkSubmitSuite, ClassLoaderIsolationSuite,
TaskSetManagerSuite (core)
- SQLQuerySuite (sql/core)
- HiveUDFDynamicLoadSuite, HiveDDLSuite, HiveQuerySuite, HiveUDFSuite,
SQLQuerySuite (sql/hive)
- CliSuite, HiveThriftServer2Suites (sql/hive-thriftserver)
- ReplE2ESuite (sql/connect)
No tests were added or removed. Existing tests now compile Java sources at
runtime instead of loading pre-built JARs.
### Was this patch authored or co-authored using generative AI tooling?
Kiro CLI / Opus 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #55192: [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries
URL: https://github.com/apache/spark/pull/55192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #55192:
URL: https://github.com/apache/spark/pull/55192#issuecomment-4191535831
Thank you @HyukjinKwon and @pan3793 for reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #55192:
URL: https://github.com/apache/spark/pull/55192#discussion_r3038117523
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3865,19 +3865,27 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
test("SPARK-33084: Add jar support Ivy URI in SQL -- jar contains udf class") {
- val jarPath = Thread.currentThread().getContextClassLoader
- .getResource("SPARK-33084.jar")
- assume(jarPath != null)
val sumFuncClass = "org.apache.spark.examples.sql.Spark33084"
+ val resourceName = "SPARK-33084/Spark33084.java"
+ val sourceUrl = Thread.currentThread().getContextClassLoader
+ .getResource(resourceName)
+ assert(sourceUrl != null, s"Resource not found: $resourceName")
+ val source = Map(sumFuncClass ->
+ new String(java.nio.file.Files.readAllBytes(
+ java.nio.file.Paths.get(sourceUrl.toURI)),
+ java.nio.charset.StandardCharsets.UTF_8))
+ val classpath = java.lang.management.ManagementFactory.getRuntimeMXBean.getClassPath
+ .split(java.io.File.pathSeparator).map(p => new java.io.File(p).toURI.toURL).toSeq
+ val jarFile = new java.io.File(Utils.createTempDir(), "SPARK-33084.jar")
+ org.apache.spark.util.SparkTestUtils.createJarWithJavaSources(source, jarFile, classpath)
Review Comment:
there are a few places use the inline package name, does AI tend to do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #55192:
URL: https://github.com/apache/spark/pull/55192#discussion_r3037930794
##########
core/src/test/scala/org/apache/spark/executor/ClassLoaderIsolationSuite.scala:
##########
@@ -20,41 +20,41 @@ package org.apache.spark.executor
import java.io.File
import java.net.URL
-import scala.util.Properties
-
import org.apache.spark.{JobArtifactSet, JobArtifactState, LocalSparkContext, SparkConf, SparkContext, SparkFunSuite, TestUtils}
import org.apache.spark.util.{MutableURLClassLoader, Utils}
class ClassLoaderIsolationSuite extends SparkFunSuite with LocalSparkContext {
- private val scalaVersion = Properties.versionNumberString
- .split("\\.")
- .take(2)
- .mkString(".")
-
- private val jarURL1 = Thread.currentThread().getContextClassLoader.getResource("TestUDTF.jar")
+ private val jarURL1 = TestUtils.createJarWithClasses(Seq("ClassLoaderIsolation_Dummy"))
private lazy val jar1 = jarURL1.toString
- // package com.example
- // object Hello { def test(): Int = 2 }
- // case class Hello(x: Int, y: Int)
- private val jarURL2 = Thread.currentThread().getContextClassLoader
- .getResource(s"TestHelloV2_$scalaVersion.jar")
+ // Dynamically generate JARs that mimic Scala's `object Hello { def test(): Int = N }`
+ // The test code accesses Hello$ class via MODULE$ field and test() method.
+ private def createHelloJar(returnValue: Int): URL = {
+ // scalastyle:off line.size.limit
+ val source = Map(
+ "com.example.Hello$" ->
+ s"""package com.example;
+ |public final class Hello$$ implements java.io.Serializable {
+ | public static final Hello$$ MODULE$$;
+ | static { MODULE$$ = new Hello$$(); }
+ | private Hello$$() {}
+ | public int test() { return $returnValue; }
Review Comment:
this looks a little bit weird, the source code should be written in scala ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #55192:
URL: https://github.com/apache/spark/pull/55192#discussion_r3038138125
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3865,19 +3865,27 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
test("SPARK-33084: Add jar support Ivy URI in SQL -- jar contains udf class") {
- val jarPath = Thread.currentThread().getContextClassLoader
- .getResource("SPARK-33084.jar")
- assume(jarPath != null)
val sumFuncClass = "org.apache.spark.examples.sql.Spark33084"
+ val resourceName = "SPARK-33084/Spark33084.java"
+ val sourceUrl = Thread.currentThread().getContextClassLoader
+ .getResource(resourceName)
+ assert(sourceUrl != null, s"Resource not found: $resourceName")
+ val source = Map(sumFuncClass ->
+ new String(java.nio.file.Files.readAllBytes(
+ java.nio.file.Paths.get(sourceUrl.toURI)),
+ java.nio.charset.StandardCharsets.UTF_8))
+ val classpath = java.lang.management.ManagementFactory.getRuntimeMXBean.getClassPath
+ .split(java.io.File.pathSeparator).map(p => new java.io.File(p).toURI.toURL).toSeq
+ val jarFile = new java.io.File(Utils.createTempDir(), "SPARK-33084.jar")
+ org.apache.spark.util.SparkTestUtils.createJarWithJavaSources(source, jarFile, classpath)
Review Comment:
Hmm, I don't know the reason why AI uses the inline package name this time but will modify them before committing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #55192:
URL: https://github.com/apache/spark/pull/55192#issuecomment-4191234770
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56353][BUILD][TESTS] Generate Java-based test JARs dynamically instead of storing pre-built binaries [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #55192:
URL: https://github.com/apache/spark/pull/55192#discussion_r3038138125
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -3865,19 +3865,27 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
test("SPARK-33084: Add jar support Ivy URI in SQL -- jar contains udf class") {
- val jarPath = Thread.currentThread().getContextClassLoader
- .getResource("SPARK-33084.jar")
- assume(jarPath != null)
val sumFuncClass = "org.apache.spark.examples.sql.Spark33084"
+ val resourceName = "SPARK-33084/Spark33084.java"
+ val sourceUrl = Thread.currentThread().getContextClassLoader
+ .getResource(resourceName)
+ assert(sourceUrl != null, s"Resource not found: $resourceName")
+ val source = Map(sumFuncClass ->
+ new String(java.nio.file.Files.readAllBytes(
+ java.nio.file.Paths.get(sourceUrl.toURI)),
+ java.nio.charset.StandardCharsets.UTF_8))
+ val classpath = java.lang.management.ManagementFactory.getRuntimeMXBean.getClassPath
+ .split(java.io.File.pathSeparator).map(p => new java.io.File(p).toURI.toURL).toSeq
+ val jarFile = new java.io.File(Utils.createTempDir(), "SPARK-33084.jar")
+ org.apache.spark.util.SparkTestUtils.createJarWithJavaSources(source, jarFile, classpath)
Review Comment:
Hmm, I don't know the reason why AI use the inline package name this time but will modify them before committing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org