[I] Enable `bloom_filter` on `parquet::arrow::ArrowWriter` increase file size by multiple times. [arrow-rs]

["nooberfsh (via GitHub)" <gi...@apache.org>]

nooberfsh opened a new issue, #4970:
URL: https://github.com/apache/arrow-rs/issues/4970

   ```rust
   use arrow_array::{ArrayRef, Int32Array, RecordBatch};
   use parquet::arrow::ArrowWriter;
   use parquet::basic::Compression;
   use parquet::file::properties::WriterProperties;
   use std::fs::File;
   use std::sync::Arc;
   
   fn main() {
       let ids = Int32Array::from(vec![1, 2, 3, 4]);
       let vals = Int32Array::from(vec![5, 6, 7, 8]);
       let batch = RecordBatch::try_from_iter(vec![
           ("id", Arc::new(ids) as ArrayRef),
           ("val", Arc::new(vals) as ArrayRef),
       ])
       .unwrap();
   
       let file = File::create("foo.txt").unwrap();
   
       // WriterProperties can be used to set Parquet file options
       let props = WriterProperties::builder()
           .set_compression(Compression::SNAPPY)
           .set_bloom_filter_enabled(true) 
           .build();
   
       let mut writer = ArrowWriter::try_new(file, batch.schema(), Some(props)).unwrap();
   
       writer.write(&batch).expect("Writing batch");
   
       // writer must be closed to write footer
       writer.close().unwrap();
   }
   
   ```
   In the above example, the file is 806Byte when  `bloom_filter` is off and 2.1M when `bloom_filter` is on, is it expected?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Enable `bloom_filter` on `parquet::arrow::ArrowWriter` increase file size by multiple times. [arrow-rs]

["nooberfsh (via GitHub)" <gi...@apache.org>]

nooberfsh commented on issue #4970:
URL: https://github.com/apache/arrow-rs/issues/4970#issuecomment-1773794055

   Thanks for the quick confirmation and the extra information.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Enable `bloom_filter` on `parquet::arrow::ArrowWriter` increase file size by multiple times. [arrow-rs]

["nooberfsh (via GitHub)" <gi...@apache.org>]

nooberfsh closed issue #4970: Enable `bloom_filter` on `parquet::arrow::ArrowWriter` increase file size by multiple times.
URL: https://github.com/apache/arrow-rs/issues/4970


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Enable `bloom_filter` on `parquet::arrow::ArrowWriter` increase file size by multiple times. [arrow-rs]

["tustvold (via GitHub)" <gi...@apache.org>]

tustvold commented on issue #4970:
URL: https://github.com/apache/arrow-rs/issues/4970#issuecomment-1773787778

   Yes, the defaults are for a reasonably sized parquet file. You can tweak the bloom filter generation with
   
   https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_bloom_filter_ndv
   
   https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_bloom_filter_fpp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org