You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "vigneshsiva11 (via GitHub)" <gi...@apache.org> on 2026/02/05 14:41:45 UTC
[PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
vigneshsiva11 opened a new pull request, #9361:
URL: https://github.com/apache/arrow-rs/pull/9361
# Which issue does this PR close?
- Refs #7973
# Rationale for this change
This PR adds regression coverage for an offset overflow panic encountered when reading Parquet files with large binary columns using the Arrow Parquet reader.
The issue occurs when a single RecordBatch contains more than 2GB of binary data and is decoded into a 'BinaryArray' using 32-bit offsets. Similar failures are observed across multiple Parquet encodings. Adding regression tests helps document the failure mode and provides a foundation for validating a follow-up fix.
# What changes are included in this PR?
This PR introduces a new test file under 'parquet/tests/arrow_reader' that adds regression tests covering large binary columns for the following Parquet encodings:
- PLAIN
- DELTA_LENGTH_BYTE_ARRAY
- DELTA_BYTE_ARRAY
- RLE_DICTIONARY
The tests construct Parquet files that exceed the 32-bit offset limit when decoded into a single Arrow 'BinaryArray' and assert successful RecordBatch decoding.
# Are these changes tested?
Yes. This PR consists entirely of regression tests.
The tests are currently marked as ignored to avoid excessive memory usage in CI and to document the existing failure mode. They are intended to be enabled once the underlying reader logic is updated to safely handle large binary data without overflowing offsets.
# Are there any user-facing changes?
No. This PR only adds tests and does not introduce any user-facing changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#discussion_r2789117140
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+/// Number of rows written
+const ROWS: usize = 1024;
+
+/// Size of each binary value (~3MB)
+/// 1024 * 3MB ≈ 3GB total → guaranteed offset overflow for i32
+const VALUE_SIZE: usize = 3 * 1024 * 1024;
+
+fn make_large_binary_array() -> ArrayRef {
+ let mut builder = BinaryBuilder::new();
+
+ for _ in 0..ROWS {
+ let data = vec![b'a'; VALUE_SIZE];
+ builder.append_value(&data);
+ }
+
+ Arc::new(builder.finish()) as ArrayRef
+}
+
+fn write_parquet_with_encoding(array: ArrayRef, encoding: Encoding) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "col",
+ DataType::Binary,
+ false,
+ )]));
+
+ let batch = RecordBatch::try_new(schema.clone(), vec![array]).unwrap();
+
+ let file = tempfile().unwrap();
+
+ let builder = WriterProperties::builder();
+ let builder = match encoding {
+ Encoding::RLE_DICTIONARY => builder.set_dictionary_enabled(true),
+ _ => builder.set_dictionary_enabled(false).set_encoding(encoding),
+ };
+
+ let props = builder.build();
+
+ let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), schema, Some(props)).unwrap();
+
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+
+ file
+}
+
+#[test]
+// Panics until https://github.com/apache/arrow-rs/issues/7973 is fixed
+#[should_panic(expected = "byte array offset overflow")]
+fn large_binary_plain_encoding_overflow() {
Review Comment:
These tests allocate/write ~3GB of binary data and currently lack `#[ignore]`, which will likely cause OOM/timeouts in CI. This also contradicts the PR description stating the tests are ignored. Consider marking these tests `#[ignore]` (or gating behind an env flag), and make the assertions reflect the intended post-fix behavior (i.e., successful `RecordBatch` decoding) rather than using `#[should_panic]`.
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,134 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+/// Number of rows written
+const ROWS: usize = 1024;
+
+/// Size of each binary value (~3MB)
+/// 1024 * 3MB ≈ 3GB total → guaranteed offset overflow for i32
+const VALUE_SIZE: usize = 3 * 1024 * 1024;
+
+fn make_large_binary_array() -> ArrayRef {
+ let mut builder = BinaryBuilder::new();
+
+ for _ in 0..ROWS {
+ let data = vec![b'a'; VALUE_SIZE];
+ builder.append_value(&data);
+ }
+
+ Arc::new(builder.finish()) as ArrayRef
+}
+
+fn write_parquet_with_encoding(array: ArrayRef, encoding: Encoding) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "col",
+ DataType::Binary,
+ false,
+ )]));
+
+ let batch = RecordBatch::try_new(schema.clone(), vec![array]).unwrap();
+
+ let file = tempfile().unwrap();
+
+ let builder = WriterProperties::builder();
+ let builder = match encoding {
+ Encoding::RLE_DICTIONARY => builder.set_dictionary_enabled(true),
+ _ => builder.set_dictionary_enabled(false).set_encoding(encoding),
+ };
+
+ let props = builder.build();
+
+ let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), schema, Some(props)).unwrap();
+
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+
+ file
+}
+
+#[test]
+// Panics until https://github.com/apache/arrow-rs/issues/7973 is fixed
+#[should_panic(expected = "byte array offset overflow")]
+fn large_binary_plain_encoding_overflow() {
Review Comment:
`#[should_panic(expected = "byte array offset overflow")]` doesn't appear to match the actual overflow error text produced by the Parquet arrow reader (e.g. `"index overflow decoding byte array"` from `parquet/src/arrow/buffer/offset_buffer.rs`). As written, these tests may fail even when the overflow occurs. If you keep a panic-based test, align the expected substring with the real message, or (preferably) assert on the returned `Err` explicitly instead of relying on `unwrap()` panics.
##########
parquet/tests/arrow_reader/mod.rs:
##########
@@ -44,6 +44,7 @@ mod bloom_filter;
mod checksum;
mod int96_stats_roundtrip;
mod io;
+mod large_string_overflow;
Review Comment:
The test module/file name (`large_string_overflow`) is misleading: this code is exercising `DataType::Binary` / `BinaryBuilder`, not strings. Renaming to something like `large_binary_overflow` (and updating the `mod` declaration) would make the purpose clearer and align with the PR title.
```suggestion
mod large_binary_overflow;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#issuecomment-3878791208
This failure looks like a transient CI issue on Windows while downloading protoc ('protoc-21.4-win64.zip' unzip error).
There are no changes in this PR related to CI or protoc. Could someone please re-run the failing job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#discussion_r2784637787
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+/// Number of rows written
+const ROWS: usize = 1024;
+
+/// Size of each binary value (~3MB)
+/// 1024 * 3MB ≈ 3GB total → guaranteed offset overflow for i32
+const VALUE_SIZE: usize = 3 * 1024 * 1024;
+
+fn make_large_binary_array() -> ArrayRef {
+ let mut builder = BinaryBuilder::new();
+
+ for _ in 0..ROWS {
+ let data = vec![b'a'; VALUE_SIZE];
+ builder.append_value(&data);
+ }
+
+ Arc::new(builder.finish()) as ArrayRef
+}
+
+fn write_parquet_with_encoding(array: ArrayRef, encoding: Encoding) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "col",
+ DataType::Binary,
+ false,
+ )]));
+
+ let batch = RecordBatch::try_new(schema.clone(), vec![array]).unwrap();
+
+ let file = tempfile().unwrap();
+
+ let builder = WriterProperties::builder();
+ let builder = match encoding {
+ Encoding::RLE_DICTIONARY => builder.set_dictionary_enabled(true),
+ _ => builder.set_dictionary_enabled(false).set_encoding(encoding),
+ };
+
+ let props = builder.build();
+
+ let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), schema, Some(props)).unwrap();
+
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+
+ file
+}
+
+#[test]
+// Panics until https://github.com/apache/arrow-rs/issues/7973 is fixed
Review Comment:
Can you please apply this same pattern to the other tests in this file (so they all use should_panic and hav a comment)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#issuecomment-3867318952
Hi @alamb, Thanks for the feedback.
I’ve updated the PR to address your comments:
1. Replaced the ignored test with #[should_panic] to explicitly assert the current failure.
2. Added a comment linking to https://github.com/apache/arrow-rs/issues/7973 explaining why the panic is expected.
3. Kept the other encoding cases as ignored regression tests for the follow-up fix.
4. Reformatted the code and confirmed CI passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#issuecomment-3879254505
Thanks @vigneshsiva11 -- I restarted the windows job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#discussion_r2769569636
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,116 @@
+use std::sync::Arc;
Review Comment:
This file is missing the Apache License header that is standard across all files in this codebase. Please add the Apache Software Foundation license header at the beginning of the file, following the same format as other test files in this directory (see parquet/tests/arrow_reader/bad_data.rs or checksum.rs for reference).
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,116 @@
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, BinaryArray, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+const ROWS: usize = 500;
+const VALUE_SIZE: usize = 5_068_563; // ~5MB per row → triggers >2GB total
+
+fn make_large_binary_array() -> ArrayRef {
+ let value = vec![b'a'; VALUE_SIZE];
+ let values: Vec<Vec<u8>> = std::iter::repeat(value)
+ .take(ROWS)
+ .collect();
+
+ Arc::new(BinaryArray::from(values)) as ArrayRef
+}
+
+fn write_parquet_with_encoding(
+ array: ArrayRef,
+ encoding: Encoding,
+) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("col", DataType::Binary, false),
+ ]));
+
+ let batch =
+ RecordBatch::try_new(schema.clone(), vec![array])
+ .unwrap();
+
+ let file = tempfile().unwrap();
+
+ let props = WriterProperties::builder()
+ .set_dictionary_enabled(true)
+ .set_encoding(encoding)
+ .build();
+
Review Comment:
The `write_parquet_with_encoding` function sets both `set_dictionary_enabled(true)` and `set_encoding(encoding)`. According to the WriterProperties documentation, when dictionary is enabled and a specific encoding is set, the specified encoding becomes the "fallback" encoding (see parquet/src/file/properties.rs:1079-1081). This means these tests may not actually be testing the specified encodings directly, but rather testing dictionary encoding with these as fallbacks.
For more accurate testing of specific encodings, consider setting `set_dictionary_enabled(false)` for non-dictionary encodings (PLAIN, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY). The dictionary test should only use `set_dictionary_enabled(true)` without calling `set_encoding`.
```suggestion
let mut builder = WriterProperties::builder();
// For non-dictionary encodings, disable dictionary and set the encoding directly.
// For dictionary encoding, enable dictionary and rely on the default dictionary behavior.
let builder = match encoding {
Encoding::RLE_DICTIONARY => builder.set_dictionary_enabled(true),
_ => builder
.set_dictionary_enabled(false)
.set_encoding(encoding),
};
let props = builder.build();
```
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,116 @@
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, BinaryArray, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+const ROWS: usize = 500;
+const VALUE_SIZE: usize = 5_068_563; // ~5MB per row → triggers >2GB total
+
+fn make_large_binary_array() -> ArrayRef {
+ let value = vec![b'a'; VALUE_SIZE];
+ let values: Vec<Vec<u8>> = std::iter::repeat(value)
+ .take(ROWS)
+ .collect();
+
+ Arc::new(BinaryArray::from(values)) as ArrayRef
+}
+
+fn write_parquet_with_encoding(
+ array: ArrayRef,
+ encoding: Encoding,
+) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("col", DataType::Binary, false),
+ ]));
+
+ let batch =
+ RecordBatch::try_new(schema.clone(), vec![array])
+ .unwrap();
+
+ let file = tempfile().unwrap();
+
+ let props = WriterProperties::builder()
+ .set_dictionary_enabled(true)
+ .set_encoding(encoding)
+ .build();
+
+ let mut writer =
+ ArrowWriter::try_new(file.try_clone().unwrap(), schema, Some(props))
+ .unwrap();
+
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+
+ file
+}
+
+#[test]
+#[ignore = "regression test for >2GB binary offset overflow"]
+fn large_binary_plain_encoding_overflow() {
+ let array = make_large_binary_array();
+ let file = write_parquet_with_encoding(array, Encoding::PLAIN);
+
+ let mut reader =
+ ParquetRecordBatchReaderBuilder::try_new(file)
+ .unwrap()
+ .build()
+ .unwrap();
+
+ assert!(matches!(reader.next(), Some(Ok(_))));
+}
+
+#[test]
+#[ignore = "regression test for >2GB binary offset overflow"]
+fn large_binary_delta_length_encoding_overflow() {
+ let array = make_large_binary_array();
+ let file = write_parquet_with_encoding(
+ array,
+ Encoding::DELTA_LENGTH_BYTE_ARRAY,
+ );
+
+ let mut reader =
+ ParquetRecordBatchReaderBuilder::try_new(file)
+ .unwrap()
+ .build()
+ .unwrap();
+
+ assert!(matches!(reader.next(), Some(Ok(_))));
+}
+
+#[test]
+#[ignore = "regression test for >2GB binary offset overflow"]
+fn large_binary_delta_byte_array_encoding_overflow() {
+ let array = make_large_binary_array();
+ let file =
+ write_parquet_with_encoding(array, Encoding::DELTA_BYTE_ARRAY);
+
+ let mut reader =
+ ParquetRecordBatchReaderBuilder::try_new(file)
+ .unwrap()
+ .build()
+ .unwrap();
+
+ assert!(matches!(reader.next(), Some(Ok(_))));
+}
+
+#[test]
+#[ignore = "regression test for >2GB binary offset overflow"]
+fn large_binary_rle_dictionary_encoding_overflow() {
+ let array = make_large_binary_array();
+ let file =
+ write_parquet_with_encoding(array, Encoding::RLE_DICTIONARY);
Review Comment:
This test will panic when run. The WriterProperties builder at line 40 calls `set_encoding(encoding)` with `Encoding::RLE_DICTIONARY`, but according to the parquet properties implementation (parquet/src/file/properties.rs:1087-1089), calling `set_encoding` with `RLE_DICTIONARY` or `PLAIN_DICTIONARY` will panic with "Dictionary encoding can not be used as fallback encoding".
To enable dictionary encoding, you should only use `set_dictionary_enabled(true)` without calling `set_encoding(Encoding::RLE_DICTIONARY)`. The dictionary encoding will be used automatically when dictionary is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on code in PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#discussion_r2788779941
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::sync::Arc;
+
+use arrow_array::builder::BinaryBuilder;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+/// Number of rows written
+const ROWS: usize = 1024;
+
+/// Size of each binary value (~3MB)
+/// 1024 * 3MB ≈ 3GB total → guaranteed offset overflow for i32
+const VALUE_SIZE: usize = 3 * 1024 * 1024;
+
+fn make_large_binary_array() -> ArrayRef {
+ let mut builder = BinaryBuilder::new();
+
+ for _ in 0..ROWS {
+ let data = vec![b'a'; VALUE_SIZE];
+ builder.append_value(&data);
+ }
+
+ Arc::new(builder.finish()) as ArrayRef
+}
+
+fn write_parquet_with_encoding(array: ArrayRef, encoding: Encoding) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "col",
+ DataType::Binary,
+ false,
+ )]));
+
+ let batch = RecordBatch::try_new(schema.clone(), vec![array]).unwrap();
+
+ let file = tempfile().unwrap();
+
+ let builder = WriterProperties::builder();
+ let builder = match encoding {
+ Encoding::RLE_DICTIONARY => builder.set_dictionary_enabled(true),
+ _ => builder.set_dictionary_enabled(false).set_encoding(encoding),
+ };
+
+ let props = builder.build();
+
+ let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), schema, Some(props)).unwrap();
+
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+
+ file
+}
+
+#[test]
+// Panics until https://github.com/apache/arrow-rs/issues/7973 is fixed
Review Comment:
Hi @alamb, thanks for the review!
I’ve updated the PR to apply the same pattern to all tests in this file:
Converted the remaining cases to use #[should_panic(expected = "byte array offset overflow")]
Added a comment on each test linking to https://github.com/apache/arrow-rs/issues/7973 to document why the panic is currently expected
Removed the ignored tests so all cases are explicitly asserted
I’ve also reformatted the code and confirmed CI passes. Please let me know if this looks good or if you’d like any further adjustments.
Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add regression tests for Parquet large binary offset overflow [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9361:
URL: https://github.com/apache/arrow-rs/pull/9361#discussion_r2779183358
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,116 @@
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, BinaryArray, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+const ROWS: usize = 500;
+const VALUE_SIZE: usize = 5_068_563; // ~5MB per row → triggers >2GB total
+
+fn make_large_binary_array() -> ArrayRef {
+ let value = vec![b'a'; VALUE_SIZE];
+ let values: Vec<Vec<u8>> = std::iter::repeat(value)
+ .take(ROWS)
+ .collect();
+
+ Arc::new(BinaryArray::from(values)) as ArrayRef
+}
+
+fn write_parquet_with_encoding(
+ array: ArrayRef,
+ encoding: Encoding,
+) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("col", DataType::Binary, false),
+ ]));
+
+ let batch =
+ RecordBatch::try_new(schema.clone(), vec![array])
+ .unwrap();
+
+ let file = tempfile().unwrap();
+
+ let props = WriterProperties::builder()
+ .set_dictionary_enabled(true)
+ .set_encoding(encoding)
+ .build();
+
Review Comment:
I think this is an important comment to resolve
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use std::sync::Arc;
+
+use arrow_array::{ArrayRef, BinaryArray, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
+use parquet::arrow::arrow_writer::ArrowWriter;
+use parquet::basic::Encoding;
+use parquet::file::properties::WriterProperties;
+
+use tempfile::tempfile;
+
+const ROWS: usize = 500;
+const VALUE_SIZE: usize = 5_068_563; // ~5MB per row → triggers >2GB total
+
+fn make_large_binary_array() -> ArrayRef {
+ let value = vec![b'a'; VALUE_SIZE];
+ let values: Vec<Vec<u8>> = std::iter::repeat(value).take(ROWS).collect();
+
+ Arc::new(BinaryArray::from(values)) as ArrayRef
+}
+
+fn write_parquet_with_encoding(array: ArrayRef, encoding: Encoding) -> std::fs::File {
+ let schema = Arc::new(Schema::new(vec![Field::new(
+ "col",
+ DataType::Binary,
+ false,
+ )]));
+
+ let batch = RecordBatch::try_new(schema.clone(), vec![array]).unwrap();
+
+ let file = tempfile().unwrap();
+
+ let props = WriterProperties::builder()
+ .set_dictionary_enabled(true)
+ .set_encoding(encoding)
+ .build();
+
+ let mut writer = ArrowWriter::try_new(file.try_clone().unwrap(), schema, Some(props)).unwrap();
+
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+
+ file
+}
+
+#[test]
+#[ignore = "regression test for >2GB binary offset overflow"]
Review Comment:
Rather than ignoring the tests, can you please
1. add an assertion that it should panic?
2. Add a comment with a link to the ticket?
That way we are sure the test properly covers what you want
For example
```rust
#[test]
// Panics until https://github.com/apache/arrow-rs/issues/7973 is fixed
#[should_panic(expected = "byte array offset overflow")]
fn large_binary_plain_encoding_overflow() {
```
##########
parquet/tests/arrow_reader/large_string_overflow.rs:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
In order for this test to actually be run, I think you need to add it to `mod.rs`
As it is the file is ignored and the tests are not run
```diff
index ffc36655b3..33fc59d494 100644
--- a/parquet/tests/arrow_reader/mod.rs
+++ b/parquet/tests/arrow_reader/mod.rs
@@ -48,6 +48,7 @@ mod io;
mod predicate_cache;
mod row_filter;
mod statistics;
+mod large_string_overflow;
// returns a struct array with columns "int32_col", "float32_col" and "float64_col" with the specified values
fn struct_array(input: Vec<(Option<i32>, Option<f32>, Option<f64>)>) -> ArrayRef {
```
When I add that this file doesn't compile
I had to change the building ot
```rust
fn make_large_binary_array() -> ArrayRef {
let value = vec![b'a'; VALUE_SIZE];
let mut builder = BinaryBuilder::new();
for _ in 0..ROWS {
builder.append_value(&value);
}
Arc::new(builder.finish())
}
```
And then it seems to work well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org