You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/07 10:05:42 UTC
[I] API to get memory usage for Arr [arrow-rs]
alamb opened a new issue, #5851:
URL: https://github.com/apache/arrow-rs/issues/5851
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When writing parquet files, depending on the writer settings and the data being written, we have observed the ArrowWriter consuming large amounts of memory (10s of GB) -- see https://github.com/apache/arrow-rs/issues/5828
The memory usage of parquet writers also often comes up in the context of proposals for new parquet formats
There is already a discussion about how to limit memory when writing here https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html#memory-limiting
However there is now way currently to get a measurement of actual current use (that we could use to abort the write, for example).
**Describe the solution you'd like**
I would like some way to get to have some visibility on the current memory usage of the internal buffering in the parquet writer
**Describe alternatives you've considered**
I propose adding a function to [ArrowWriter](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html#) modeled on [Array::get_array_memory_size](https://docs.rs/arrow/latest/arrow/array/trait.Array.html#tymethod.get_array_memory_size)
```rust
impl ArrayWriter {
/// returns an estimate of how much memory the array
/// writer is currently using in its internal buffers.
fn memory_size(&self) -> usize { ... }
...
}
```
**Additional context**
Here is one ticket that describes one non trivial source of memory usage https://github.com/apache/arrow-rs/issues/5828 so the indices should be included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #5851:
URL: https://github.com/apache/arrow-rs/issues/5851#issuecomment-2185532223
> I'll be on the lookout for a PR -- please ping me when you are ready for feedback
ok!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #5851:
URL: https://github.com/apache/arrow-rs/issues/5851#issuecomment-2185093656
can I take it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #5851:
URL: https://github.com/apache/arrow-rs/issues/5851#issuecomment-2197868792
> @wiedld made a PR for this feature: #5967 as well
ok, planned to code it in weekend, so still no codes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5851:
URL: https://github.com/apache/arrow-rs/issues/5851#issuecomment-2185476385
I'll be on the lookout for a PR -- please ping me when you are ready for feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5851:
URL: https://github.com/apache/arrow-rs/issues/5851#issuecomment-2185186474
That would be amazing -- thank you @Rachelint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #5851: API to get memory usage for parquet ArrowWriter
URL: https://github.com/apache/arrow-rs/issues/5851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] API to get memory usage for parquet ArrowWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5851:
URL: https://github.com/apache/arrow-rs/issues/5851#issuecomment-2154964775
BTE there is already https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html#method.in_progress_size but that only accounts for the actual parquet data in progress, not any internal buffering structures in the writer itself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org