[I] parquet: panic (BitChunks "offset + len out of bounds") reading definition levels with a bit-packed run longer than the page data [arrow-rs]

["techgopal (via GitHub)" <gi...@apache.org>]

techgopal opened a new issue, #11691:
URL: https://github.com/apache/arrow-rs/issues/11691

   ### Describe the bug
   
   Reading a nullable column whose definition levels contain a bit-packed run that is longer than the remaining level data panics in `arrow-buffer` instead of returning an error.
   
   `PackedDecoder::next_rle_block` (`parquet/src/arrow/record_reader/definition_levels.rs`) sets `packed_count` from the run header, and `PackedDecoder::read` then calls
   
   ```rust
   buffer.append_packed_range(offset..offset + to_read, self.data.as_ref());
   ```
   
   without checking that `self.data` holds `ceil((offset + to_read) / 8)` bytes. `BooleanBufferBuilder::append_packed_range` documents that it panics in that case, and it does, from `BitChunks::new`:
   
   ```
   assert!(ceil(end, 8) <= buffer.len(), "offset + len out of bounds");
   ```
   
   This is different from #11516 / #11517 (overflow of the run length): here the run length is small and valid, it just exceeds the bytes available in the page.
   
   ### To Reproduce
   
   With `parquet = "60.0.0"`, `arrow-array = "60.0.0"`, `arrow-schema = "60.0.0"`, `bytes = "1"`:
   
   ```rust
   use std::sync::Arc;
   
   use arrow_array::{Int32Array, RecordBatch};
   use arrow_schema::{DataType, Field, Schema};
   use bytes::Bytes;
   use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
   use parquet::arrow::ArrowWriter;
   use parquet::file::properties::WriterProperties;
   
   fn main() {
       // A nullable INT32 column with every other value null: the data page
       // (v1, uncompressed, plain) starts with the definition levels as
       // [4-byte length = 127][bit-packed header 0x7F (63 groups = 504 levels)][63 bytes]...
       let array: Int32Array = (0..1000).map(|i| (i % 2 == 0).then_some(i)).collect();
       let schema = Arc::new(Schema::new(vec![Field::new("c", DataType::Int32, true)]));
       let batch = RecordBatch::try_new(schema.clone(), vec![Arc::new(array)]).unwrap();
       let props = WriterProperties::builder().set_dictionary_enabled(false).build();
       let mut buf = Vec::new();
       let mut writer = ArrowWriter::try_new(&mut buf, schema, Some(props)).unwrap();
       writer.write(&batch).unwrap();
       writer.close().unwrap();
   
       // Shrink the definition-level length prefix from 127 to 3 bytes; the
       // first bit-packed run header still claims 504 levels.
       let pattern = [0x7F, 0, 0, 0, 0x7F, 0x55];
       let pos = buf.windows(pattern.len()).position(|w| w == pattern).expect("pattern");
       buf[pos] = 3;
   
       let reader = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(buf))
           .unwrap()
           .build()
           .unwrap();
       for batch in reader {
           println!("{:?}", batch.map(|b| b.num_rows()));
       }
   }
   ```
   
   Output (debug and release):
   
   ```
   thread 'main' panicked at arrow-buffer-60.0.0/src/util/bit_chunk_iterator.rs:235:9:
   offset + len out of bounds
   ```
   
   Backtrace (abridged):
   
   ```
   arrow_buffer::util::bit_chunk_iterator::BitChunks::new
   arrow_buffer::util::bit_util::byte_aligned_bitwise_bin_op_helper
   arrow_buffer::util::bit_util::apply_bitwise_binary_op
   arrow_buffer::builder::boolean::BooleanBufferBuilder::append_packed_range
   parquet::arrow::record_reader::definition_levels::PackedDecoder::read
   <parquet::arrow::record_reader::definition_levels::DefinitionLevelBufferDecoder as DefinitionLevelDecoder>::read_def_levels
   parquet::column::reader::GenericColumnReader<R,D,V>::read_records_with_reservation
   ...
   ```
   
   The same panic is reachable through `ParquetPushDecoder::try_decode`.
   
   ### Expected behavior
   
   The reader returns a `ParquetError` (e.g. unexpected end of definition level data) instead of panicking.
   
   ### Additional context
   
   - Reproduces with `parquet` / `arrow-buffer` 60.0.0 and on `main` at 33d338ddf (`definition_levels.rs:446`).
   - Found while fuzzing a Parquet reader built on arrow-rs. Related: #11516, #9742.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org