You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zhangxinyao88 (via GitHub)" <gi...@apache.org> on 2026/09/11 00:43:31 UTC

[PR] PARQUET: Account for custom PageIndexProvider memory [arrow-rs]

zhangxinyao88 opened a new pull request, #11057:
URL: https://github.com/apache/arrow-rs/pull/11057

   # Which issue does this PR close?
   
   - Closes #11028.
   
   # Rationale for this change
   
   `ParquetMetaData::memory_size()` currently accounts for page-index memory only when the provider is the built-in `PageIndex`. Custom `PageIndexProvider` implementations are therefore reported as zero even when they own cached index data.
   
   # What changes are included in this PR?
   
   - Add a default `PageIndexProvider::heap_size() -> usize` method returning zero.
   - Have the built-in `PageIndex` delegate to its existing private `HeapSize` implementation.
   - Account for the concrete provider size, `Arc` allocation overhead, and provider-reported heap allocation in `ParquetMetaData::memory_size()`.
   - Add regression coverage for a custom provider that reports heap usage.
   
   # Are these changes tested?
   
   - `cargo test -p parquet file::metadata --no-default-features`
   - `cargo check -p parquet --example custom_page_index --no-default-features --features arrow`
   - `rustfmt --check` on modified files.
   
   This change was developed with AI assistance. I reviewed the implementation and verified the built-in accounting path, the custom-provider regression test, and source compatibility for the existing custom-provider example.
   
   # Are there any user-facing changes?
   
   Yes. Custom `PageIndexProvider` implementations can opt into `ParquetMetaData::memory_size()` accounting by overriding `heap_size()`. Existing implementations remain source-compatible and report zero additional heap usage by default.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET: Account for custom PageIndexProvider memory [arrow-rs]

Posted by "zhangxinyao88 (via GitHub) via github" <gi...@arrow.apache.org>.
zhangxinyao88 commented on PR #11057:
URL: https://github.com/apache/arrow-rs/pull/11057#issuecomment-6939483258

   Could we keep this PR open as a draft while we settle the design in #11028? I’m holding off on implementation changes until we agree on how custom providers can reuse the existing heap-size helpers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET: Account for custom PageIndexProvider memory [arrow-rs]

Posted by "github-actions[bot] (via GitHub) via github" <gi...@arrow.apache.org>.
github-actions[bot] commented on PR #11057:
URL: https://github.com/apache/arrow-rs/pull/11057#issuecomment-6811841457

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET: Account for custom PageIndexProvider memory [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11057:
URL: https://github.com/apache/arrow-rs/pull/11057#issuecomment-5629057310

   Thanks @zhangxinyao88, but I think a PR for this issue is premature. I think we first need to decide on the approach we want to take before we start implementing solutions. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] PARQUET: Account for custom PageIndexProvider memory [arrow-rs]

Posted by "zhangxinyao88 (via GitHub)" <gi...@apache.org>.
zhangxinyao88 commented on PR #11057:
URL: https://github.com/apache/arrow-rs/pull/11057#issuecomment-5646291697

   Thanks, that makes sense. I’ll mark this as draft until we’ve agreed on an approach in #11028.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org