You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "databird07 (via GitHub)" <gi...@apache.org> on 2026/10/16 04:10:18 UTC
[PR] chore(ci): build Hudi once per profile set and reuse it in the GitHub test jobs [hudi]
databird07 opened a new pull request, #20363:
URL: https://github.com/apache/hudi/pull/20363
### Describe the issue this Pull Request addresses
Every test job in `bot.yml` runs its own `mvn clean install -DskipTests` before testing. In run 38362333401 the 11 builds that share a profile set took 63.5 of ~622 job-minutes.
### Summary and Changelog
- Three build jobs (JDK11/Spark 3.5, Java 17/Spark 4.2, Flink) save their module `target/` directories and `~/.m2/.../org/apache/hudi` as a workflow artifact (pax tar + zstd, duplicate jars hard-linked; ~1 GB).
- Test jobs that use the same profiles restore that artifact instead of building. Restored timestamps are re-stamped (`compress_mtimes.py`) so Maven treats the outputs as up to date. Test steps themselves are unchanged.
- Bundle validation, Spark 4 bundle validation and integration tests keep their own builds because they use different profiles.
### Impact
CI-only. Projected saving is ~30 job-minutes per push. The critical path gets ~3 minutes longer, because test jobs wait for a build job. Locally, 20 runs each of in-job build vs. restored artifact gave identical test counts and identical JaCoCo coverage, with 0 recompiled sources after restore. This hasn't run on GitHub Actions yet, so this PR's own CI run is the first real check.
### Risk Level
medium. Workflow restructuring that depends on the restored outputs being treated as up to date. Mitigated by manifest checks (commit and profile set) on restore and by the local comparison above. Reverting is a single-commit revert.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DNM] chore(ci): build Hudi once per profile set and reuse it in the GitHub test jobs [hudi]
Posted by "databird07 (via GitHub)" <gi...@apache.org>.
databird07 closed pull request #20363: [DNM] chore(ci): build Hudi once per profile set and reuse it in the GitHub test jobs
URL: https://github.com/apache/hudi/pull/20363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DNM] chore(ci): build Hudi once per profile set and reuse it in the GitHub test jobs [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20363:
URL: https://github.com/apache/hudi/pull/20363#issuecomment-6208983096
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20363?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.08%. Comparing base ([`e9da226`](https://app.codecov.io/gh/apache/hudi/commit/e9da226db3a90747cc51d13fec56ebd1704c1699?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fbd45ed`](https://app.codecov.io/gh/apache/hudi/commit/fbd45ed4612330a4753b268d9861db588cba3696?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 18 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20363 +/- ##
============================================
- Coverage 80.70% 80.08% -0.62%
- Complexity 35124 35185 +61
============================================
Files 2553 2553
Lines 143524 144918 +1394
Branches 17474 17740 +266
============================================
+ Hits 115830 116060 +230
- Misses 19746 20889 +1143
- Partials 7948 7969 +21
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.64% <ø> (-0.51%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.94% <ø> (ø)` | |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.96% <ø> (-1.10%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.64% <ø> (-1.55%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.17% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.86% <ø> (-0.30%)` | :arrow_down: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.79% <ø> (-1.80%)` | :arrow_down: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.83% <ø> (+0.30%)` | :arrow_up: |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20363/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `51.88% <ø> (-2.60%)` | :arrow_down: |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20363/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration-tests](https://app.codecov.io/gh/apache/hudi/pull/20363/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `13.45% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 25 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20363/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DNM] chore(ci): build Hudi once per profile set and reuse it in the GitHub test jobs [hudi]
Posted by "databird07 (via GitHub)" <gi...@apache.org>.
databird07 commented on PR #20363:
URL: https://github.com/apache/hudi/pull/20363#issuecomment-6208217340
Combined into #20364 so the total impact can be reviewed in one place; this commit is included there unchanged. Closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org