You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/06/04 17:03:29 UTC

[I] Bound ArrowWriter peak memory by spilling buffered Parquet pages (pluggable PageStore) [arrow-rs]

alamb opened a new issue, #10071:
URL: https://github.com/apache/arrow-rs/issues/10071

   ### Is your feature request related to a problem?
   `ArrowWriter` must buffer every column's *compressed* pages for an entire row group before it can splice them into contiguous column chunks at flush, so peak memory is ≈ Σ(compressed bytes of all column chunks) and grows with row group size. On wide, skewed schemas (e.g. ~400 columns, some columns far larger than others) this can consume >=12 GB of memory just to write Parquet. The only existing lever is flushing smaller row groups via `in_progress_size`, which trades away compression and read-time page/row-group pruning.
   
   ### Describe the solution you'd like
   Introduce a pluggable buffering trait (`PageStore`) so completed pages can be spilled (e.g. to disk) instead of held in memory, with the in-memory default preserving today's behavior and byte-for-byte output. The writer should stream each page back out of the store one at a time at splice rather than materializing the whole chunk, bounding peak write memory by the in-flight encoder/dictionary buffers instead of the row group size.
   
   ### Describe alternatives you've considered
   Reducing row group size to limit buffering, but this sacrifices encoding efficiency and read performance and doesn't address the underlying coupling where one column's size forces the page layout of others.
   
   ### Additional context
   Related issues:
   - #5828
   - #5450
   - #5484
   - Fixed by PR #10020.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Bound ArrowWriter peak memory [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #10071: Bound ArrowWriter peak memory 
URL: https://github.com/apache/arrow-rs/issues/10071


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Bound ArrowWriter peak memory [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10071:
URL: https://github.com/apache/arrow-rs/issues/10071#issuecomment-4625356770

   I made a demo that shows how effective the memory savings from this approach can be:
   https://github.com/alamb/wide-parquet
   
   <img width="894" height="387" alt="Image" src="https://github.com/user-attachments/assets/e8fea9e4-9b2d-4154-bfb9-51a7eee85804" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Bound ArrowWriter peak memory [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10071:
URL: https://github.com/apache/arrow-rs/issues/10071#issuecomment-4624750219

   `label_issue.py` automatically added labels {'parquet'} from #10020


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org