You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/10/09 08:29:42 UTC

[PR] perf(spark): stop deep schema comparisons per record in partial-update merging [hudi]

yihua opened a new pull request, #20269:
URL: https://github.com/apache/hudi/pull/20269

   ### Describe the issue this Pull Request addresses
   
   closes #20268
   
   Merging a partial update into a full record built the merged schema by converting the Spark struct type back to a `HoodieSchema`, which drops field defaults (for example the `-1` default of the position-merging row index column). That schema is unequal to the reader schema but has the same Avro hash, so every schema-keyed lookup on the merge path (`LocalHoodieSchemaCache`, `HoodieInternalRowUtils.getCachedSchema`, the merged-schema cache, `isPartial`) compared the two field by field once per merged record. #12949 removed this class of cost by interning schemas; interning does not help here because the two schemas are not equal.
   
   ### Summary and Changelog
   
   When the merged fields are all the reader fields, `SparkRecordMergingUtils.getCachedMergedSchema` now returns the reader schema itself; partial-only merges are unchanged. `HoodieSchema.hashCode` delegates to the Avro schema's cached hash instead of `Objects.hash`. New `TestSparkRecordMergingUtils` covers the full-plus-partial, full-update and partial-plus-partial merges (two cases fail on master).
   
   ### Impact
   
   Lower CPU and allocation when reading or compacting MOR file groups with partial-update log records on Spark. Single-thread merge benchmark (31-field reader schema, 2-column partial records): 2367 to 486 ns and 13.8 KB to 0.7 KB allocated per merged record. Merged values are unchanged.
   
   ### Risk Level
   
   low. A full newer record is now returned as is instead of copied, so in event-time ordering `deltaMerge` returns empty (keep the buffered record) when the buffered full record wins; `TestBufferedRecordMerger` is updated for that. The kept record has the same values, ordering value and delete flag, and keeps its own operation and key. `TestPartialUpdateForMergeInto` passes.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(spark): stop deep schema comparisons per record in partial-update merging [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20269:
URL: https://github.com/apache/hudi/pull/20269#issuecomment-6077692049

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20269?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 80.62%. Comparing base ([`308f78e`](https://app.codecov.io/gh/apache/hudi/commit/308f78e71deabf7a9eec4b0a2ea26b1360658030?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d606f5e`](https://app.codecov.io/gh/apache/hudi/commit/d606f5e39b2da377438d5f4c278b1396076096b7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff            @@
   ##             master   #20269   +/-   ##
   =========================================
     Coverage     80.62%   80.62%           
     Complexity    35055    35055           
   =========================================
     Files          2552     2552           
     Lines        143400   143398    -2     
     Branches      17454    17454           
   =========================================
     Hits         115615   115615           
   + Misses        19867    19866    -1     
   + Partials       7918     7917    -1     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.16% <100.00%> (ø)` | |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.82% <ø> (+<0.01%)` | :arrow_up: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.96% <ø> (ø)` | |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.05% <ø> (ø)` | |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.19% <ø> (ø)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.11% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.18% <ø> (ø)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.43% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20269/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/20269/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20269/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.99% <100.00%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20269?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...org/apache/hudi/merge/SparkRecordMergingUtils.java](https://app.codecov.io/gh/apache/hudi/pull/20269?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmerge%2FSparkRecordMergingUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWVyZ2UvU3BhcmtSZWNvcmRNZXJnaW5nVXRpbHMuamF2YQ==) | `97.01% <ø> (-0.09%)` | :arrow_down: |
   | [...va/org/apache/hudi/common/schema/HoodieSchema.java](https://app.codecov.io/gh/apache/hudi/pull/20269?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fschema%2FHoodieSchema.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3NjaGVtYS9Ib29kaWVTY2hlbWEuamF2YQ==) | `87.42% <100.00%> (ø)` | |
   
   ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20269/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(spark): stop deep schema comparisons per record in partial-update merging [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #20269: perf(spark): stop deep schema comparisons per record in partial-update merging
URL: https://github.com/apache/hudi/pull/20269


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] perf(spark): stop deep schema comparisons per record in partial-update merging [hudi]

Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20269:
URL: https://github.com/apache/hudi/pull/20269#issuecomment-6701681909

   Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org