You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/09/12 17:36:38 UTC
[PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
voonhous opened a new pull request, #19928:
URL: https://github.com/apache/hudi/pull/19928
### Describe the issue this Pull Request addresses
Part of #19524, stacked on the shared-session and state-scoping PRs (successors of #19921 and #19923). Supersedes the proof of concept #19925. The Java CI wall clock is set by the Scala SQL shards, and scalatest has no fork count, so the lever is running suites concurrently in one JVM. This PR turns that on for the three SQL-only shards and measures it.
### Summary and Changelog
- Pom properties `hudi.scalatest.parallel` (default false) and `hudi.scalatest.threads` (default 0) mapped to the scalatest plugin's `parallel` and `threadCount`: suites run concurrently, tests within a suite stay sequential. Set to true and 2 on dml-1, dml-2 and other-1, next to the shared-session flag they carry.
- Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's name prefix or as a session temp view (derived names such as `<generated>_pt` were dropped by other suites), and two helpers that sorted log file paths as strings now sort by the log file comparator (the write token precedes the instant in the name, so stage ids compared as text). The second is latent on master.
Measurement, over three runs of each of the three shards: the scalatest "Run completed" wall clock against the same job serial. dml-1 and other-1 are the two jobs setting the run's critical runtime of about 41 minutes under #19892, so this is where the wall clock should move. A result inside the serial range is no gain.
| shard | serial under #19892 (its measured runs) | sample 1 | sample 2 | sample 3 |
|---|---|---|---|---|
| dml-1 (dml.others) | about 30 min of tests | pending | pending | pending |
| dml-2 (dml.insert, dml.schema) | about 23 | pending | pending | pending |
| other-1 (procedure, ddl) | about 32 | pending | pending | pending |
| run's longest job | 41 to 44 | pending | pending | pending |
<details>
<summary>Evidence from the proof of concept (#19925)</summary>
On master's layout the single dml job (dml-1 plus dml-2 together) at two threads: 27:58, 35:27 and 35:32 ([run 34697891947](https://github.com/apache/hudi/actions/runs/34697891947), attempts 1 to 3) against 41:22 on master's fastest runner and 52 to 56 on its typical one, the same 326 test names each time, both threads busy 96 to 98% of the time. Locally, twenty iterations of twelve interaction-heavy suites at two threads were green, and the whole dml tree at two threads passed in 22 minutes.
</details>
### Impact
CI configuration and test infrastructure only.
### Risk Level
low. Off by default; on three shards whose suites were prepared for it in the two PRs below and looped locally at two threads.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5647967500
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba4b6460734c95a21324a69f4af19a3891c4f949 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056)
* 3aab942ccdcb90e5733548c108c6a0c9cfcd8685 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5651677019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
}, {
"hash" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066",
"triggerID" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"triggerType" : "PUSH"
}, {
"hash" : "fe4e20376ae5cca827b29f17653bd953ba56a281",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17067",
"triggerID" : "fe4e20376ae5cca827b29f17653bd953ba56a281",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8abd450c3c4bc716caaf00b54d4a315b713f3d8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066)
* fe4e20376ae5cca827b29f17653bd953ba56a281 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17067)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5652180188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
}, {
"hash" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066",
"triggerID" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"triggerType" : "PUSH"
}, {
"hash" : "fe4e20376ae5cca827b29f17653bd953ba56a281",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17067",
"triggerID" : "fe4e20376ae5cca827b29f17653bd953ba56a281",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fe4e20376ae5cca827b29f17653bd953ba56a281 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17067)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5647583949
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba4b6460734c95a21324a69f4af19a3891c4f949 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5647615580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba4b6460734c95a21324a69f4af19a3891c4f949 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5649197300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3aab942ccdcb90e5733548c108c6a0c9cfcd8685 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5651652655
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
}, {
"hash" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066",
"triggerID" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"triggerType" : "PUSH"
}, {
"hash" : "fe4e20376ae5cca827b29f17653bd953ba56a281",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fe4e20376ae5cca827b29f17653bd953ba56a281",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8abd450c3c4bc716caaf00b54d4a315b713f3d8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066)
* fe4e20376ae5cca827b29f17653bd953ba56a281 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5648832812
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba4b6460734c95a21324a69f4af19a3891c4f949 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056)
* 3aab942ccdcb90e5733548c108c6a0c9cfcd8685 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): run SQL suites two at a time on three shards [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19928:
URL: https://github.com/apache/hudi/pull/19928#discussion_r4127272720
##########
pom.xml:
##########
@@ -220,6 +220,12 @@
<skipUTs>${skipTests}</skipUTs>
<skipFTs>${skipTests}</skipFTs>
<skipITs>${skipTests}</skipITs>
+ <!-- Surefire forks per module, expressed per CPU core the way Maven -T is, so a 4-core GitHub
+ runner gets two forks and a 2-core Azure agent gets one; each fork gets the full argLine
+ heap; override with -Dhudi.surefire.forkCount=1 to serialise a module (for example when
+ bisecting a test that depends on shared state), and pass an explicit integer on machines
+ with a single core. -->
+ <hudi.surefire.forkCount>0.5C</hudi.surefire.forkCount>
Review Comment:
🤖 Since this default lives in the root pom, it also applies to every local `mvn test`/`mvn install`. On a 12-core laptop that's 6 forks, each with `-Xmx4g` plus its own Spark/Hive/Kafka services, and the functional profiles no longer pin `forkCount=1`. Would it be safer to keep the pom default at `1` and pass `-Dhudi.surefire.forkCount=0.5C` (or `2`) from the CI workflows? This may belong to the base PR (#19892) rather than this one.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-hadoop-common/src/main/java/org/apache/hudi/common/util/ParquetUtils.java:
##########
@@ -350,9 +350,26 @@ public HoodieFileFormat getFormat() {
*/
@Override
public List<GenericRecord> readAvroRecords(HoodieStorage storage, StoragePath filePath) {
+ // A schema-less read takes the schema from the file; the copy drops any read schema or
+ // projection a previous caller left on the shared configuration.
+ Configuration conf = storage.getConf().unwrapCopyAs(Configuration.class);
+ conf.unset("parquet.avro.read.schema");
Review Comment:
🤖 This protects `readAvroRecords` from the leaked schema, but the leak itself is still there: `HDFSParquetImporterUtils#buildHoodieRecordsForImport` (line 185) writes `parquet.avro.read.schema` onto `jsc.hadoopConfiguration()`. That happens after `Job.getInstance` has already copied the conf, so the import job never sees it. Would it be worth dropping that write (or moving it onto `job.getConfiguration()`) so the procedure stops mutating the SparkContext-wide conf?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/testutils/HiveTestService.java:
##########
@@ -63,7 +63,9 @@
public class HiveTestService {
private static final int CONNECTION_TIMEOUT_MS = 30000;
private static final String BIND_HOST = "127.0.0.1";
- private static final int HS2_THRIFT_PORT = 9999;
+ // Chosen once per JVM rather than fixed at 9999 so that surefire forks running this module concurrently
+ // each get their own HiveServer2 port; within one JVM every HiveTestService instance still shares the port, as before.
+ private static final int HS2_THRIFT_PORT = NetworkTestUtils.nextFreePort();
Review Comment:
🤖 Choosing this once at class init keeps HS2 (and ZK, same pattern) on one ephemeral-range port for the whole reused fork. Between classes, while HS2 is stopped, the other fork's Spark/Kafka/`nextFreePort()` binds can take it and the next `start()` fails to bind; the metastore `nextFreePort()` in `configureHive` can even return this same port. Could the port be picked per `start()` like the metastore port, with callers using `getJdbcHive2Url()` instead of the static `HS2_JDBC_URL`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/insert/TestInsertTable4.scala:
##########
@@ -290,10 +290,22 @@ class TestInsertTable4 extends HoodieSparkSqlTestBase {
var listenerCallCount: Int = 0
var countDownLatch: CountDownLatch = _
- // add a listener for stages for parallelism checking with stage name
- class StageParallelismListener(var stageName: String) extends SparkListener {
+ // add a listener for stages for parallelism checking with stage name, restricted to the stages of
+ // the jobs this test started. Hudi overwrites the job group id and description on its own jobs, so
+ // the discriminator is a local property the test sets before the query.
+ class StageParallelismListener(var stageName: String, scope: String) extends SparkListener {
+ private val scopedStageIds = ConcurrentHashMap.newKeySet[Int]()
+
+ override def onJobStart(jobStart: SparkListenerJobStart): Unit = {
+ if (jobStart.properties != null
+ && scope == jobStart.properties.getProperty(TestInsertTable4.SCOPE_PROPERTY)) {
+ jobStart.stageIds.foreach(stageId => scopedStageIds.add(stageId))
+ }
+ }
+
override def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit = {
Review Comment:
🤖 nit: the SCOPE_PROPERTY/UUID scoping plus manual set/clear of the local property in try/finally is duplicated almost verbatim across two test methods — worth factoring into a small helper so future edits to one copy don't silently diverge from the other.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/HoodieSparkSqlTestBase.scala:
##########
@@ -71,25 +82,56 @@ class HoodieSparkSqlTestBase extends FunSuite with BeforeAndAfterAll {
// is consistent with the fixtures
DateTimeZone.setDefault(DateTimeZone.UTC)
TimeZone.setDefault(DateTimeUtils.getTimeZone("UTC"))
- protected lazy val spark: SparkSession = SparkSession.builder()
- .config("spark.sql.warehouse.dir", sparkWareHouse.getCanonicalPath)
- .config("spark.sql.session.timeZone", "UTC")
- .config("hoodie.insert.shuffle.parallelism", "4")
- .config("hoodie.upsert.shuffle.parallelism", "4")
- .config("hoodie.delete.shuffle.parallelism", "4")
- .config(sparkConf())
- .getOrCreate()
+ protected lazy val spark: SparkSession = if (sharedSessionEnabled) {
+ val session = HoodieSparkSqlTestBase.sharedBaseSession().newSession()
+ SparkSession.setActiveSession(session)
+ applySuiteConfToSharedSession(session)
+ session
+ } else {
+ HoodieSparkSqlTestBase.sessionBuilder(sparkWareHouse, sparkConf()).getOrCreate()
+ }
private var tableId = new AtomicInteger(0)
private var extraConf = Map[String, String]()
Review Comment:
🤖 nit: `hadoopConfOverrides` is a `var Seq` mutated via `:+=` and later reversed — might read more clearly as an `ArrayBuffer` given it's accumulating state to undo later.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/common/HoodieSparkSqlTestBase.scala:
##########
@@ -71,25 +82,56 @@ class HoodieSparkSqlTestBase extends FunSuite with BeforeAndAfterAll {
// is consistent with the fixtures
DateTimeZone.setDefault(DateTimeZone.UTC)
TimeZone.setDefault(DateTimeUtils.getTimeZone("UTC"))
- protected lazy val spark: SparkSession = SparkSession.builder()
- .config("spark.sql.warehouse.dir", sparkWareHouse.getCanonicalPath)
- .config("spark.sql.session.timeZone", "UTC")
- .config("hoodie.insert.shuffle.parallelism", "4")
- .config("hoodie.upsert.shuffle.parallelism", "4")
- .config("hoodie.delete.shuffle.parallelism", "4")
- .config(sparkConf())
- .getOrCreate()
+ protected lazy val spark: SparkSession = if (sharedSessionEnabled) {
+ val session = HoodieSparkSqlTestBase.sharedBaseSession().newSession()
+ SparkSession.setActiveSession(session)
+ applySuiteConfToSharedSession(session)
+ session
+ } else {
+ HoodieSparkSqlTestBase.sessionBuilder(sparkWareHouse, sparkConf()).getOrCreate()
+ }
private var tableId = new AtomicInteger(0)
private var extraConf = Map[String, String]()
+ // Shared mode: spark.hadoop.* keys this suite set on the shared Hadoop conf, with the value they replaced.
+ private var hadoopConfOverrides: Seq[(String, String)] = Seq.empty
+
def sparkConf(): SparkConf = {
val conf = getSparkConfForTest("Hoodie SQL Test")
conf.setAll(extraConf)
conf
}
+ /**
+ * Shared mode: the context-level SparkConf is fixed, so the deltas a suite adds through extraConf or a
+ * sparkConf() override go to its session conf (hoodie.* and spark.sql.* keys), except spark.hadoop.*
+ * keys, which the write client reads from sparkContext.hadoopConfiguration and which are restored in
+ * afterAll. Any other spark.* key, and any static SQL conf, is a setting a child session cannot change,
+ * so it is rejected before anything is mutated: a partial apply would leave the shared Hadoop conf
+ * changed for every later suite in the JVM.
+ */
+ private def applySuiteConfToSharedSession(session: SparkSession): Unit = {
+ val defaults = getSparkConfForTest("Hoodie SQL Test").getAll.toMap
+ val deltas = sparkConf().getAll.filterNot { case (k, v) => defaults.get(k).contains(v) }
+ val (hadoopKeys, sessionKeys) = deltas.partition { case (k, _) => k.startsWith("spark.hadoop.") }
+ val rejected = sessionKeys.collect {
+ case (k, _) if (k.startsWith("spark.") && !k.startsWith("spark.sql.")) || SQLConf.isStaticConfigKey(k) => k
+ }
+ if (rejected.nonEmpty) {
+ throw new IllegalArgumentException(
+ s"${rejected.mkString(", ")}: SparkContext-level or static settings; shared session mode cannot apply them per suite")
+ }
+ val hadoopConf = session.sparkContext.hadoopConfiguration
+ hadoopKeys.foreach { case (k, v) =>
Review Comment:
🤖 The save/restore here is only safe if suites that set `spark.hadoop.*` never overlap. If two non-exclusive suites set the same key, the first `afterAll` unsets the second suite's value mid-run, and the second then restores the first suite's value for the rest of the JVM. Could we fail fast when `hadoopKeys.nonEmpty` and the suite isn't an `ExclusiveSuite`, so a future suite can't hit this silently?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5651146796
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
}, {
"hash" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066",
"triggerID" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3aab942ccdcb90e5733548c108c6a0c9cfcd8685 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060)
* b8abd450c3c4bc716caaf00b54d4a315b713f3d8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5651126168
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
}, {
"hash" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3aab942ccdcb90e5733548c108c6a0c9cfcd8685 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060)
* b8abd450c3c4bc716caaf00b54d4a315b713f3d8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5651628696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17060",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
}, {
"hash" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066",
"triggerID" : "b8abd450c3c4bc716caaf00b54d4a315b713f3d8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8abd450c3c4bc716caaf00b54d4a315b713f3d8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17066)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5647613865
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19928?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `75.00000%` with `3 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.64%. Comparing base ([`648996f`](https://app.codecov.io/gh/apache/hudi/commit/648996f002c3c9e24a1e2bf53f97fe03e8bc4c81?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ba4b646`](https://app.codecov.io/gh/apache/hudi/commit/ba4b6460734c95a21324a69f4af19a3891c4f949?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19928?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...org/apache/hudi/index/HoodieInMemoryHashIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19928?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FHoodieInMemoryHashIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2luZGV4L0hvb2RpZUluTWVtb3J5SGFzaEluZGV4LmphdmE=) | 75.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19928?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19928 +/- ##
============================================
- Coverage 80.20% 79.64% -0.57%
- Complexity 34661 34830 +169
============================================
Files 2546 2546
Lines 142452 143912 +1460
Branches 17330 17617 +287
============================================
+ Hits 114260 114615 +355
- Misses 20281 21344 +1063
- Partials 7911 7953 +42
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.49% <ø> (-0.35%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.68% <75.00%> (-0.60%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.57% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.63% <ø> (-0.65%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.11% <ø> (-3.06%)` | :arrow_down: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.99% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.76% <ø> (-0.05%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.99% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.52% <ø> (-0.10%)` | :arrow_down: |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.34% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `80.99% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19928/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19928/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19928/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.82% <75.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19928?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...org/apache/hudi/index/HoodieInMemoryHashIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19928?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FHoodieInMemoryHashIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2luZGV4L0hvb2RpZUluTWVtb3J5SGFzaEluZGV4LmphdmE=) | `89.47% <75.00%> (-7.67%)` | :arrow_down: |
... and [86 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19928/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(spark): [DNM] run SQL suites two at a time on three shards [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19928:
URL: https://github.com/apache/hudi/pull/19928#issuecomment-5647935614
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056",
"triggerID" : "ba4b6460734c95a21324a69f4af19a3891c4f949",
"triggerType" : "PUSH"
}, {
"hash" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3aab942ccdcb90e5733548c108c6a0c9cfcd8685",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba4b6460734c95a21324a69f4af19a3891c4f949 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17056)
* 3aab942ccdcb90e5733548c108c6a0c9cfcd8685 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org