You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/08/28 04:19:51 UTC

[PR] Fix OneSideHashJoiner memory size accounting [datafusion]

kosiew opened a new pull request, #24744:
URL: https://github.com/apache/datafusion/pull/24744

   ## Which issue does this PR close?
   
   * Part of #23393
   
   ## Rationale for this change
   
   `OneSideHashJoiner::size()` can overcount memory by adding inline fields that are already included in the joiner descriptor, while its previous `HashSet<usize>` estimate only accounted for element payload capacity and omitted hashbrown control/metadata allocation.
   
   This change defines the component-level accounting contract more explicitly so the joiner descriptor is counted once and directly owned container allocations are included using their actual allocation-aware helpers where available.
   
   ## What changes are included in this PR?
   
   * Document the `OneSideHashJoiner::size()` contract as including the joiner descriptor, directly owned container allocations, and input arrays, while excluding allocations referenced by shared `on` expressions.
   * Remove double counting of inline fields already covered by `size_of_val(self)`.
   * Account for the `on` vector's owned backing allocation using its capacity.
   * Adjust `PruningJoinHashMap` accounting so its descriptor is not counted twice.
   * Replace the `visited_rows` payload-only capacity calculation with `allocation_size()`, which accounts for the hashbrown allocation layout, including control bytes.
   * Add regression coverage for empty and grown `visited_rows` states and for the owned `on`, hash map, and hashes-buffer allocations.
   * No transformer composition or shared-record-batch accounting is changed.
   
   ## Are these changes tested?
   
   Yes. This PR adds the focused unit test:
   
   `one_side_hash_joiner_size_counts_descriptor_and_hash_set_allocation_once`
   
   The test checks the explicit expected size formula for an empty joiner, verifies `visited_rows` allocation accounting across capacity growth, and verifies that owned container allocation changes are reflected without double-counting descriptors.
   
   ## Are there any user-facing changes?
   
   No public API or query-result behavior changes are introduced.
   
   This changes internal memory-size accounting for `OneSideHashJoiner`, making its reported size more consistent with the documented ownership contract.
   
   ## LLM-generated code disclosure
   
   This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix OneSideHashJoiner memory size accounting [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #24744:
URL: https://github.com/apache/datafusion/pull/24744


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Fix OneSideHashJoiner memory size accounting [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24744:
URL: https://github.com/apache/datafusion/pull/24744#issuecomment-5448531881

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24744?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 81.47%. Comparing base ([`d1fc4b3`](https://app.codecov.io/gh/apache/datafusion/commit/d1fc4b334a209cb9fb5c8b16372682ba7763a410?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f24bc2c`](https://app.codecov.io/gh/apache/datafusion/commit/f24bc2cd44e6db5bc1dbfbe9c5fddbd694f34238?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 1 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@           Coverage Diff           @@
   ##             main   #24744   +/-   ##
   =======================================
     Coverage   81.47%   81.47%           
   =======================================
     Files        1122     1122           
     Lines      403582   403675   +93     
     Branches   403582   403675   +93     
   =======================================
   + Hits       328821   328900   +79     
   - Misses      55512    55518    +6     
   - Partials    19249    19257    +8     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24744?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org