You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/10/15 17:55:46 UTC

[PR] feat(release): add scale-based release candidate validation script and skill [hudi]

nsivabalan opened a new pull request, #20352:
URL: https://github.com/apache/hudi/pull/20352

   ### Describe the issue this Pull Request addresses
   
   Validating a release candidate before voting is a manual, multi-hour checklist today. `scripts/release/validate_staged_release.sh` covers the source release, but everything after it is done by hand on every RC: downloading the staged bundles, running a quickstart, standing up a HoodieStreamer pipeline to confirm commits, compaction and cleaning, building from source, and choosing and running tests. That makes RC validation slow to start, hard to repeat, and uneven across voters. Several environment traps also produce failures that look like RC problems when they aren't:
   - tests run on a different JDK than the release builds with
   - MOR test tables so small that compaction never runs
   - spark-shell scripts that hang when a line throws
   
   ### Summary and Changelog
   
   Voters get one command that validates an RC at the depth they have time for, against the artifacts actually being voted on, with a PASS/FAIL line per check and a summary at the end.
   
   ```bash
   scripts/release/validate_rc.sh --release=1.1.2 --rc_num=1 --scale=minimal|decent|large \
     --staging_repo=<staging repo url> --signing_key=<key id> --spark_home=<spark 3.5 dir>
   ```
   
   | Scale | Steps | Time |
   |---|---|---|
   | minimal | source release checks (now including signing-key match); quickstart on COW and MOR (datasource + Spark SQL, 21 checks) against the staged spark bundle; continuous HoodieStreamer run on MOR | ~5-10 min |
   | decent | minimal + build from the voted source tarball with the JDK its pom declares + HoodieStreamer tests + a larger HoodieStreamer run | ~15-30 min |
   | large | decent + `TestCOWDataSource`, `TestMORDataSource`, `TestInsertTable`, `TestMergeIntoTable` + high-volume HoodieStreamer runs on MOR and COW | ~2-4 h |
   
   Changelog:
   - `scripts/release/validate_rc.sh`: entry point. Picks steps from `--scale`; `--steps` overrides. Each step runs isolated, so one failure doesn't hide the others' results.
   - `scripts/release/rc_validation/`:
     - `common.sh`: picks the JDK from the source pom and downloads staged bundles with sha1 checks. Its spark-shell runner treats compile errors and REPL fall-through as failures.
     - `quickstart.scala`: the COW and MOR quickstart checks.
     - `run_streamer_e2e.sh`, `streamer_e2e_datagen.scala`, `streamer_e2e_validate.scala`, `trips.avsc`: HoodieStreamer in continuous mode with `ParquetDFSSource` and `FilebasedSchemaProvider`, fed by a timed batch generator. Checks commit count, error records, ERROR log lines, record count against the expected total, duplicate keys, arrival of the last batch, completed async compaction (MOR) and files deleted by the cleaner.
     - `run_tests.sh`: scale-based test selection. It fails if no tests ran, so a selector that matches nothing doesn't produce a green build.
   - `scripts/release/skills/hudi-rc-validation/SKILL.md`: an agent skill. It maps a request like "quick sanity" or "thorough" to a scale, runs the script, separates known environment failures from real RC problems, and drafts the vote reply. Coding agents can use it; people can follow it as a runbook.
   
   Verified against 1.1.2-rc1 (`orgapachehudi-1198`):
   - minimal: all steps pass, ~8.5 min
   - decent: all steps pass, ~28 min; 280k-record MOR run, build from source tarball on JDK 8, 10/10 HoodieStreamer tests
   - COW e2e at minimal size: passes
   - large: not run end to end. Its pieces are the same scripts at higher volume, plus suites that passed separately (243 datasource tests, 38 Spark SQL tests).
   
   While building this I found a pre-existing bug, which I'll file separately: `CALL show_cleans(..., showArchived => true)` fails with `Not an Avro data file` once cleans are archived to the LSM timeline. It affects the release as well as master.
   
   ### Impact
   
   New release tooling only, under `scripts/release/`. No change to any module, public API, config or build.
   
   ### Risk Level
   
   none. The scripts are run manually by release voters and nothing in the build or CI calls them. All new files carry ASF headers and pass `apache-rat:check`.
   
   ### Documentation Update
   
   none. Usage is documented in the `validate_rc.sh` header (`--help`) and in the skill. A follow-up can link it from the release guide on the website.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(release): add scale-based release candidate validation script and skill [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20352:
URL: https://github.com/apache/hudi/pull/20352#issuecomment-6701732906

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(release): add scale-based release candidate validation script and skill [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #20352:
URL: https://github.com/apache/hudi/pull/20352#discussion_r4286684203


##########
scripts/release/rc_validation/quickstart.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Quickstart-style sanity checks for a release candidate, run through spark-shell -i.
+// Covers COW and MOR via the datasource API and Spark SQL. Prints "QS PASS|FAIL <check>"
+// lines and exits non-zero if any check fails.
+// Required conf: spark.hudi.rc.quickstart.dir (local working directory).
+
+import org.apache.spark.sql.SaveMode._
+import org.apache.spark.sql.functions._
+
+val workDir = spark.conf.get("spark.hudi.rc.quickstart.dir")

Review Comment:
   We should reuse the docker demo scripts or consolidate them into a single one so that both the docker demo and the release validation can use the same, instead of maintaining multiple scripts achieving the same functionality.



##########
scripts/release/validate_rc.sh:
##########
@@ -0,0 +1,192 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# End-to-end validation of a Hudi release candidate before voting, at a chosen scale:
+#
+#   minimal (~5-10 min)  source release checks, quickstart on the staged bundles, short HoodieStreamer
+#                        continuous run on MOR
+#   decent  (~15-30 min) minimal + build from the source release + HoodieStreamer tests + a larger
+#                        HoodieStreamer run
+#   large   (~2-4 h)     decent + Spark datasource and Spark SQL test suites + high-volume
+#                        HoodieStreamer runs on both MOR and COW

Review Comment:
   Are these scales for functionality and performance? For performance, we should track the numbers to identify regression.



##########
scripts/release/rc_validation/streamer_e2e_datagen.scala:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Feeds a ParquetDFSSource directory with parquet batches at a fixed interval, run through spark-shell -i.
+// Batch 0 inserts `initial` keys; every later batch upserts up to `updates` existing keys and inserts
+// `inserts` new keys. Each batch is staged elsewhere and then renamed into the source directory so the
+// streamer never sees a partially written batch. The expected final key count is written to
+// <dir>/expected_total.
+// Required confs: spark.hudi.rc.e2e.{dir,batches,initial,updates,inserts,interval.seconds}
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.spark.sql.functions._
+
+def conf(key: String) = spark.conf.get(s"spark.hudi.rc.e2e.$key")

Review Comment:
   We should consider using #12196 and [Lake Loader](https://github.com/onehouseinc/lake-loader) to align the setup, which covers more data types and workloads; and incorporate proper scale testing as part of the release certification, instead of adding new script.
   
   Also, are the scenarios added in this PR already covered by the bundle validation as part of the [release candidate validation](https://github.com/apache/hudi/blob/master/.github/workflows/release_candidate_validation.yml)?  If not, could you add the cases to the existing bundle validation to avoid duplication?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(release): add scale-based release candidate validation script and skill [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20352:
URL: https://github.com/apache/hudi/pull/20352#issuecomment-6200791648

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20352?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 80.73%. Comparing base ([`5dee638`](https://app.codecov.io/gh/apache/hudi/commit/5dee63857193d19e5a61110ed6bbbbffe7955094?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5a01108`](https://app.codecov.io/gh/apache/hudi/commit/5a011082c76090cf82c603f089f8395f70a5b359?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##             master   #20352   +/-   ##
   =========================================
     Coverage     80.73%   80.73%           
     Complexity    35183    35183           
   =========================================
     Files          2553     2553           
     Lines        143596   143596           
     Branches      17496    17496           
   =========================================
     Hits         115938   115938           
     Misses        19695    19695           
     Partials       7963     7963           
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.17% <ø> (ø)` | |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.96% <ø> (ø)` | |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.09% <ø> (ø)` | |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.18% <ø> (ø)` | |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.14% <ø> (ø)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.18% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.65% <ø> (ø)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.52% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20352/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.90% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/20352/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20352/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.95% <ø> (ø)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(release): add scale-based release candidate validation script and skill [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #20352:
URL: https://github.com/apache/hudi/pull/20352#discussion_r4286622420


##########
scripts/release/rc_validation/streamer_e2e_datagen.scala:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Feeds a ParquetDFSSource directory with parquet batches at a fixed interval, run through spark-shell -i.
+// Batch 0 inserts `initial` keys; every later batch upserts up to `updates` existing keys and inserts
+// `inserts` new keys. Each batch is staged elsewhere and then renamed into the source directory so the
+// streamer never sees a partially written batch. The expected final key count is written to
+// <dir>/expected_total.
+// Required confs: spark.hudi.rc.e2e.{dir,batches,initial,updates,inserts,interval.seconds}
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.spark.sql.functions._
+
+def conf(key: String) = spark.conf.get(s"spark.hudi.rc.e2e.$key")
+val dir = conf("dir")
+val numBatches = conf("batches").toInt
+val initial = conf("initial").toLong
+val updates = conf("updates").toLong
+val inserts = conf("inserts").toLong
+val intervalMs = conf("interval.seconds").toLong * 1000
+
+val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
+var nextKey = 0L
+
+for (b <- 0 until numBatches) {
+  val start = System.currentTimeMillis()
+  val ids = if (b == 0) {
+    spark.range(0, initial).toDF("id")
+  } else {
+    val updated = spark.range(0, updates).select(pmod(hash(lit(b), col("id")), lit(nextKey)).cast("long").as("id")).distinct()
+    updated.union(spark.range(nextKey, nextKey + inserts).toDF("id"))

Review Comment:
   🤖 nit: could you add a short comment explaining that this picks pseudo-random existing keys via a hash of the batch number, and that `distinct` means fewer than `updates` keys may be updated?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
scripts/release/rc_validation/quickstart.scala:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Quickstart-style sanity checks for a release candidate, run through spark-shell -i.
+// Covers COW and MOR via the datasource API and Spark SQL. Prints "QS PASS|FAIL <check>"
+// lines and exits non-zero if any check fails.
+// Required conf: spark.hudi.rc.quickstart.dir (local working directory).
+
+import org.apache.spark.sql.SaveMode._
+import org.apache.spark.sql.functions._
+
+val workDir = spark.conf.get("spark.hudi.rc.quickstart.dir")
+val base = s"file://$workDir/tables"

Review Comment:
   🤖 non-blocking: A relative `--work_dir` (e.g. `--work_dir=rc-work`) makes this `file://rc-work/...`, and Hadoop then reads `rc-work` as the authority and throws `Wrong FS`. Quickstart and the e2e run both build `file://$DIR` paths, so both would fail in a way that looks like an RC problem. Could `validate_rc.sh` make the path absolute right after `mkdir -p`, e.g. `WORK_DIR=$(cd "$WORK_DIR" && pwd)`?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
scripts/release/rc_validation/streamer_e2e_datagen.scala:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Feeds a ParquetDFSSource directory with parquet batches at a fixed interval, run through spark-shell -i.
+// Batch 0 inserts `initial` keys; every later batch upserts up to `updates` existing keys and inserts
+// `inserts` new keys. Each batch is staged elsewhere and then renamed into the source directory so the
+// streamer never sees a partially written batch. The expected final key count is written to
+// <dir>/expected_total.
+// Required confs: spark.hudi.rc.e2e.{dir,batches,initial,updates,inserts,interval.seconds}
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.spark.sql.functions._
+
+def conf(key: String) = spark.conf.get(s"spark.hudi.rc.e2e.$key")
+val dir = conf("dir")
+val numBatches = conf("batches").toInt
+val initial = conf("initial").toLong
+val updates = conf("updates").toLong
+val inserts = conf("inserts").toLong
+val intervalMs = conf("interval.seconds").toLong * 1000
+
+val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
+var nextKey = 0L
+
+for (b <- 0 until numBatches) {
+  val start = System.currentTimeMillis()
+  val ids = if (b == 0) {
+    spark.range(0, initial).toDF("id")
+  } else {
+    val updated = spark.range(0, updates).select(pmod(hash(lit(b), col("id")), lit(nextKey)).cast("long").as("id")).distinct()
+    updated.union(spark.range(nextKey, nextKey + inserts).toDF("id"))
+  }

Review Comment:
   🤖 nit: `nextKey` is advanced after it is used as the upper bound for updates and the start of inserts, so the order matters; could you move the update to after `val df` or add a comment to make that clear?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
scripts/release/rc_validation/streamer_e2e_datagen.scala:
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Feeds a ParquetDFSSource directory with parquet batches at a fixed interval, run through spark-shell -i.
+// Batch 0 inserts `initial` keys; every later batch upserts up to `updates` existing keys and inserts
+// `inserts` new keys. Each batch is staged elsewhere and then renamed into the source directory so the
+// streamer never sees a partially written batch. The expected final key count is written to
+// <dir>/expected_total.
+// Required confs: spark.hudi.rc.e2e.{dir,batches,initial,updates,inserts,interval.seconds}
+
+import org.apache.hadoop.fs.{FileSystem, Path}
+import org.apache.spark.sql.functions._
+
+def conf(key: String) = spark.conf.get(s"spark.hudi.rc.e2e.$key")
+val dir = conf("dir")
+val numBatches = conf("batches").toInt
+val initial = conf("initial").toLong
+val updates = conf("updates").toLong
+val inserts = conf("inserts").toLong
+val intervalMs = conf("interval.seconds").toLong * 1000
+
+val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
+var nextKey = 0L
+
+for (b <- 0 until numBatches) {
+  val start = System.currentTimeMillis()
+  val ids = if (b == 0) {
+    spark.range(0, initial).toDF("id")
+  } else {
+    val updated = spark.range(0, updates).select(pmod(hash(lit(b), col("id")), lit(nextKey)).cast("long").as("id")).distinct()
+    updated.union(spark.range(nextKey, nextKey + inserts).toDF("id"))
+  }
+  nextKey = if (b == 0) initial else nextKey + inserts
+  val df = ids.select(
+    format_string("key-%012d", col("id")).as("uuid"),
+    lit(start).as("ts"),
+    lit(s"rider-$b").as("rider"),
+    lit(s"driver-$b").as("driver"),
+    (rand(b) * 100).as("fare"),
+    lit(b).as("batch"),
+    when(col("id") % 3 === 0, "san_francisco").when(col("id") % 3 === 1, "sao_paulo").otherwise("chennai").as("city"))
+  val staged = s"$dir/staging/batch_$b"
+  val batchSize = if (b == 0) initial else updates + inserts
+  df.repartition(math.max(1L, batchSize / 500000L).toInt).write.mode("overwrite").parquet(staged)
+  val files = fs.listStatus(new Path(staged)).map(_.getPath).filter(_.getName.endsWith(".parquet"))
+  files.zipWithIndex.foreach { case (p, i) => fs.rename(p, new Path(f"$dir/source/batch_$b%04d_$i%03d.parquet")) }

Review Comment:
   🤖 non-blocking: When a batch has more than one file (batch 0 at `large`), a streamer sync that runs between these renames checkpoints the first file's mtime. `DFSPathSelector` only takes files with mtime > checkpoint, so if the second file was written earlier it is skipped for good and the count check fails spuriously. Renaming the whole staged directory into `source/` would make each batch appear at once, since the selector recurses and ignores `_SUCCESS` and `.crc`:
   ```scala
   fs.rename(new Path(staged), new Path(f"$dir/source/batch_$b%04d"))
   ```
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(release): add scale-based release candidate validation script and skill [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #20352: feat(release): add scale-based release candidate validation script and skill
URL: https://github.com/apache/hudi/pull/20352


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org