You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kszlim (via GitHub)" <gi...@apache.org> on 2024/12/05 21:48:13 UTC
[I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
kszlim opened a new issue, #6839:
URL: https://github.com/apache/arrow-rs/issues/6839
**Describe the bug**
The arrow writer doesn't track memory size correctly, and it seems like it thinks `FixedSizeList` columns have a fixed memory usage. Ie. the reported memory usage doesn't grow despite the buffers actually growing in memory.
**To Reproduce**
```toml
[package]
name = "repro"
version = "0.1.0"
edition = "2021"
[dependencies]
arrow = "53.3.0"
parquet = "53.3.0"
rand = "0.8.5"
```
```rust
use arrow::array::{FixedSizeListBuilder, UInt8Builder};
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use parquet::arrow::ArrowWriter;
use parquet::file::properties::WriterProperties;
use rand::Rng;
use std::fs::File;
use std::sync::Arc;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Define the field and schema for a single column that is a fixed-size list of floats.
let list_length = 1_048_576;
let field = Field::new(
"mylist",
DataType::FixedSizeList(Arc::new(Field::new("item", DataType::UInt8, true)), list_length),
true,
);
let schema = Arc::new(Schema::new(vec![field]));
// Create a writer for the Parquet file
let file = File::create("output_randomized.parquet")?;
let props = WriterProperties::builder().build();
let mut writer = ArrowWriter::try_new(file, schema.clone(), Some(props))?;
let iterations = 10000;
let values_per_batch = list_length;
let mut list_arr_builder = FixedSizeListBuilder::new(UInt8Builder::new(), list_length);
for _ in 0..iterations {
// Generate random data for the values array
let mut rng = rand::thread_rng();
let values: Vec<u8> = (0..values_per_batch)
.map(|_| rng.gen())
.collect();
list_arr_builder.values().append_slice(&values);
list_arr_builder.append(true);
let output = list_arr_builder.finish();
let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(output)])?;
let in_memory_size = writer.memory_size() + writer.in_progress_size();
let before_in_memory_size_mb = (in_memory_size as f64) / (1024f64.powi(2));
writer.write(&batch)?;
let in_memory_size = writer.memory_size() + writer.in_progress_size();
let after_in_memory_size_mb = (in_memory_size as f64) / (1024f64.powi(2));
let change_in_usage = before_in_memory_size_mb - after_in_memory_size_mb;
dbg!(change_in_usage, after_in_memory_size_mb, before_in_memory_size_mb);
}
writer.close()?;
println!("Wrote 10000 record batches with randomized data to output_randomized.parquet");
Ok(())
}
```
**Expected behavior**
We should see the reported memory usage rise over time, then as flush is triggered, it should go down to around zero. Then repeat.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6839:
URL: https://github.com/apache/arrow-rs/issues/6839#issuecomment-2528465530
> Unfortunately I've been swamped and probably don't have time to fix it @alamb , I hope my reproduction will be enough for someone to pick it up!
100% -- much appreciated 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6839:
URL: https://github.com/apache/arrow-rs/issues/6839#issuecomment-2549320456
`label_issue.py` automatically added labels {'parquet'} from #6862
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #6839: parquet arrow writer doesn't track memory size correctly for fixed sized lists
URL: https://github.com/apache/arrow-rs/issues/6839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6839:
URL: https://github.com/apache/arrow-rs/issues/6839#issuecomment-2521731031
The issue is that GenericColumnWriter::memory_size is not accounting for the data_pages it has buffered waiting for the dictionary page to be flushed. This should be a relatively straightforward case of changing it to be
```
pub(crate) fn memory_size(&self) -> usize {
self.data_pages.iter().map(|x| x.data().len()).sum::<usize>()
+ self.column_metrics.total_bytes_written as usize
+ self.encoder.estimated_memory_size()
}
```
And adding an appropriate test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
Posted by "kszlim (via GitHub)" <gi...@apache.org>.
kszlim commented on issue #6839:
URL: https://github.com/apache/arrow-rs/issues/6839#issuecomment-2525289581
Unfortunately I've been swamped and probably don't have time to fix it @alamb , I hope my reproduction will be enough for someone to pick it up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] parquet arrow writer doesn't track memory size correctly for fixed sized lists [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6839:
URL: https://github.com/apache/arrow-rs/issues/6839#issuecomment-2525103817
Thanks @kszlim and @tustvold
@kszlim are you planning to propose a fix for this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org