[GitHub] [arrow-rs] tustvold commented on a diff in pull request #3102: parquet bloom filter part II: read sbbf bitset from row group reader, update API, and add cli demo

[GitBox <gi...@apache.org>]

tustvold commented on code in PR #3102:
URL: https://github.com/apache/arrow-rs/pull/3102#discussion_r1022016903


##########
parquet/src/bin/parquet-show-bloom-filter.rs:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Binary file to read bloom filter data from a Parquet file.
+//!
+//! # Install
+//!
+//! `parquet-show-bloom-filter` can be installed using `cargo`:
+//! ```
+//! cargo install parquet --features=cli
+//! ```
+//! After this `parquet-show-bloom-filter` should be available:
+//! ```
+//! parquet-show-bloom-filter --file-name XYZ.parquet --column id --values a
+//! ```
+//!
+//! The binary can also be built from the source code and run as follows:
+//! ```
+//! cargo run --features=cli --bin parquet-show-bloom-filter -- --file-name XYZ.parquet --column id --values a
+//! ```
+
+extern crate parquet;

Review Comment:
   ```suggestion
   ```



##########
parquet/src/bloom_filter/mod.rs:
##########
@@ -79,6 +81,33 @@ fn block_check(block: &Block, hash: u32) -> bool {
 /// A split block Bloom filter
 pub struct Sbbf(Vec<Block>);
 
+// this size should not be too large to not to hit short read too early (although unlikely)
+// but also not to small to ensure cache efficiency, this is essential a "guess" of the header
+// size. In the demo test the size is 15 bytes.
+const STEP_SIZE: usize = 16;

Review Comment:
   Based on the thrift compact protocol encoding - https://raw.githubusercontent.com/apache/thrift/master/doc/specs/thrift-compact-protocol.md.
   
   The BloomFilterHeader consists of 1 int32 and 3 enums. Enumerations are encoded as int32. Each int32 is encoded as 1 to 5 bytes.
   
   Therefore the maximum header size is 20 bytes. One possibility might be to read 20 bytes, and bail on error?
   
   



##########
parquet/src/bloom_filter/mod.rs:
##########
@@ -79,6 +81,33 @@ fn block_check(block: &Block, hash: u32) -> bool {
 /// A split block Bloom filter
 pub struct Sbbf(Vec<Block>);
 
+// this size should not be too large to not to hit short read too early (although unlikely)
+// but also not to small to ensure cache efficiency, this is essential a "guess" of the header
+// size. In the demo test the size is 15 bytes.
+const STEP_SIZE: usize = 16;
+
+/// given an initial offset, and a chunk reader, try to read out a bloom filter header by trying
+/// one or more iterations, returns both the header and the offset after it (for bitset).
+fn chunk_read_bloom_filter_header_and_offset<R: ChunkReader>(
+    offset: usize,
+    reader: Arc<R>,
+) -> Result<(BloomFilterHeader, usize), ParquetError> {
+    let mut length = STEP_SIZE;
+    loop {
+        let buffer = reader.get_bytes(offset as u64, length)?;
+        let mut buf_reader = buffer.reader();
+        let mut prot = TCompactInputProtocol::new(&mut buf_reader);
+        if let Ok(h) = BloomFilterHeader::read_from_in_protocol(&mut prot) {
+            let buffer = buf_reader.into_inner();
+            let bitset_offset = offset + length - buffer.remaining();
+            return Ok((h, bitset_offset));
+        } else {
+            // continue to try by reading another batch

Review Comment:
   Corrupt data I think will cause this to potentially iterate indefinitely, which would be bad...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org