You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/15 20:30:50 UTC

[GitHub] [arrow] sfc-gh-nthimmegowda opened a new pull request, #14147: ARROW-17450 : Support RLE encoding for boolean datatype

sfc-gh-nthimmegowda opened a new pull request, #14147:
URL: https://github.com/apache/arrow/pull/14147

   ARROW-17450
   
   Currently, parquet-cpp does not support columns encoded with RLE encoding. Although the users of RLE encoding are quite sparse with uses of one of the 3 types [Repetition and definition levels, dictionary indices and boolean values in data pages], some implementations do encode this directly on boolean columns (Athena on AWS). Even though there is encoding and decoding support for repetition and definition levels, there is no support for boolean column with RLE encoding. 
   
   This PR integrates the column scanning to support columns with RLE encoding. The first 4 bytes of the data length are size of the encoded data, which is parsed first and then passes to decoder. 
   
   Added a test with rle boolean encoded parquet file to validate that values can be parsed individually and in batch. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977215744


##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));
+
+    // For this file, 3rd index value is null
+    if (i == 2) {
+      ASSERT_TRUE(is_null);
+    } else {
+      ASSERT_FALSE(is_null);
+    }
+  }
+
+  ASSERT_FALSE(scanner->HasNext());
+  ASSERT_FALSE(scanner->NextValue(&val, &is_null));
+}
+
+TEST_F(TestBooleanRLE, TestBatchRead) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto col = std::dynamic_pointer_cast<BoolReader>(group->Column(0));
+
+  // This file only has 8 rows
+  ASSERT_EQ(8, reader_->metadata()->num_rows());
+  // This file only has 1 row group
+  ASSERT_EQ(1, reader_->metadata()->num_row_groups());
+  // Size of the metadata is 106 bytes
+  ASSERT_EQ(106, reader_->metadata()->size());
+  // This row group must have 8 rows
+  ASSERT_EQ(8, group->metadata()->num_rows());
+
+  // Check if the column is encoded with RLE
+  auto col_chunk = group->metadata()->ColumnChunk(0);
+  ASSERT_TRUE(std::find(col_chunk->encodings().begin(), col_chunk->encodings().end(),
+                        Encoding::RLE) != col_chunk->encodings().end());
+
+  // Assert column has values to be read
+  ASSERT_TRUE(col->HasNext());
+  int64_t curr_batch_read = 0;
+
+  const int16_t batch_size = 8;
+  int16_t def_levels[batch_size];
+  int16_t rep_levels[batch_size];
+  bool values[batch_size - 1];
+
+  auto levels_read =
+      col->ReadBatch(batch_size, def_levels, rep_levels, values, &curr_batch_read);
+  ASSERT_EQ(batch_size, levels_read);
+
+  // Since one value is a null value, expect batches read to be one less than indicated
+  // batch_size
+  ASSERT_EQ(batch_size - 1, curr_batch_read);
+
+  // 3rd index is null value
+  ASSERT_THAT(def_levels, testing::ElementsAre(1, 1, 0, 1, 1, 1, 1, 1));
+
+  // Validate inserted data is as expected
+  ASSERT_THAT(values, testing::ElementsAre(1, 0, 1, 1, 0, 0, 1));

Review Comment:
   OK. Could you update the test file in https://github.com/apache/parquet-testing and then update the `cpp/submodules/parquet-testing/` in this pull request?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974762118


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");

Review Comment:
   Updated to show invalid position



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r973778431


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;

Review Comment:
   `bool`?
   
   Or can we use `decoder_->GetBatch()` in this method too?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");

Review Comment:
   Can we also show invalid position?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;

Review Comment:
   ```suggestion
       auto decoder_data = data + 4;
   ```



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {

Review Comment:
   ```suggestion
       if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
   ```



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) {
+    max_values = std::min(max_values, num_values_);
+    if (!decoder_->GetBatch(buffer, max_values)) {
+      ParquetException::EofException();
+    }
+
+    num_values_ -= max_values;

Review Comment:
   It seems that `decoder_->GetBatch()` returns the number of read values. It may be smaller than `max_values`. So this may cause invalid `num_values_`.



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) {

Review Comment:
   `override` is missing?
   
   https://github.com/apache/arrow/actions/runs/3076096805/jobs/4970024700#step:6:1903
   
   ```text
   FAILED: src/parquet/CMakeFiles/parquet_objlib.dir/encoding.cc.o 
   /usr/bin/ccache /usr/lib/ccache/clang++-12  -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -DARROW_NO_DEPRECATED_API -DARROW_UBSAN -DARROW_WITH_BROTLI -DARROW_WITH_BZ2 -DARROW_WITH_LZ4 -DARROW_WITH_RE2 -DARROW_WITH_SNAPPY -DARROW_WITH_UTF8PROC -DARROW_WITH_ZLIB -DARROW_WITH_ZSTD -DBOOST_ALL_NO_LIB -DHAVE_INTTYPES_H -DHAVE_NETDB_H -DHAVE_NETINET_IN_H -DPARQUET_EXPORTING -DPARQUET_THRIFT_VERSION_MAJOR=0 -DPARQUET_THRIFT_VERSION_MINOR=13 -DUTF8PROC_STATIC -Isrc -I/arrow/cpp/src -I/arrow/cpp/src/generated -isystem /arrow/cpp/thirdparty/flatbuffers/include -isystem /arrow/cpp/thirdparty/hadoop/include -isystem google_cloud_cpp_ep-install/include -isystem absl_ep-install/include -isystem crc32c_ep-install/include -isystem opentelemetry_ep-install/include -isystem protobuf_ep-install/include -isystem utf8proc_ep-install/include -isystem xsimd_ep/src/xsimd_ep-install/include -Qunused-arguments -fcolor-diagnos
 tics -ggdb -O0  -Wall -Wextra -Wdocumentation -Wshorten-64-to-32 -Wno-missing-braces -Wno-unused-parameter -Wno-constant-logical-operand -Wno-return-stack-address -Werror -Wno-unknown-warning-option -Wno-pass-failed -msse4.2  -fsanitize=address -DADDRESS_SANITIZER -fsanitize=undefined -fno-sanitize=alignment,vptr,function,float-divide-by-zero -fno-sanitize-recover=all -fsanitize-coverage=pc-table,inline-8bit-counters,edge,no-prune,trace-cmp,trace-div,trace-gep -fsanitize-blacklist=/arrow/cpp/build-support/sanitizer-disallowed-entries.txt -g -fPIC   -fsanitize-coverage=pc-table,inline-8bit-counters,edge,no-prune,trace-cmp,trace-div,trace-gep -pthread -std=c++17 -MD -MT src/parquet/CMakeFiles/parquet_objlib.dir/encoding.cc.o -MF src/parquet/CMakeFiles/parquet_objlib.dir/encoding.cc.o.d -o src/parquet/CMakeFiles/parquet_objlib.dir/encoding.cc.o -c /arrow/cpp/src/parquet/encoding.cc
   /arrow/cpp/src/parquet/encoding.cc:2405:7: error: 'Decode' overrides a member function but is not marked 'override' [-Werror,-Winconsistent-missing-override]
     int Decode(uint8_t* buffer, int max_values) {
         ^
   /arrow/cpp/src/parquet/encoding.h:400:15: note: overridden virtual function is here
     virtual int Decode(uint8_t* buffer, int max_values) = 0;
                 ^
   1 error generated.
   ```



##########
cpp/src/parquet/level_comparison.cc:
##########
@@ -44,10 +44,10 @@ struct GreaterThanDynamicFunction {
   using FunctionType = decltype(&GreaterThanBitmap);
 
   static std::vector<std::pair<DispatchLevel, FunctionType>> implementations() {
-    return {
-      { DispatchLevel::NONE, standard::GreaterThanBitmapImpl }
+    return {{DispatchLevel::NONE, standard::GreaterThanBitmapImpl}
 #if defined(ARROW_HAVE_RUNTIME_AVX2)
-      , { DispatchLevel::AVX2, GreaterThanBitmapAvx2 }
+            ,
+            {DispatchLevel::AVX2, GreaterThanBitmapAvx2}

Review Comment:
   Hmm, you might not use `clang-format-12`:
   
   https://github.com/apache/arrow/actions/runs/3076096806/jobs/4970024641#step:5:869
   
   ```text
   FAILED: CMakeFiles/check-format 
   cd /tmp/arrow-lint-mbv5v4zf/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/run_clang_format.py --clang_format_binary /usr/bin/clang-format-12 --exclude_globs /arrow/cpp/build-support/lint_exclusions.txt --source_dir /arrow/cpp/src --source_dir /arrow/cpp/examples --source_dir /arrow/cpp/tools --quiet
   
   --- /arrow/cpp/src/arrow/util/bpacking.cc
   +++ /arrow/cpp/src/arrow/util/bpacking.cc (after clang format)
   @@ -153,14 +153,13 @@
      using FunctionType = decltype(&unpack32_default);
    
      static std::vector<std::pair<DispatchLevel, FunctionType>> implementations() {
   -    return {{DispatchLevel::NONE, unpack32_default}
   +    return {
   +      { DispatchLevel::NONE, unpack32_default }
    #if defined(ARROW_HAVE_RUNTIME_AVX2)
   -            ,
   -            {DispatchLevel::AVX2, unpack32_avx2}
   +      , { DispatchLevel::AVX2, unpack32_avx2 }
    #endif
    #if defined(ARROW_HAVE_RUNTIME_AVX512)
   -            ,
   -            {DispatchLevel::AVX512, unpack32_avx512}
   +      , { DispatchLevel::AVX512, unpack32_avx512 }
    #endif
        };
      }
   
   --- /arrow/cpp/src/parquet/level_comparison.cc
   +++ /arrow/cpp/src/parquet/level_comparison.cc (after clang format)
   @@ -44,10 +44,10 @@
      using FunctionType = decltype(&GreaterThanBitmap);
    
      static std::vector<std::pair<DispatchLevel, FunctionType>> implementations() {
   -    return {{DispatchLevel::NONE, standard::GreaterThanBitmapImpl}
   +    return {
   +      { DispatchLevel::NONE, standard::GreaterThanBitmapImpl }
    #if defined(ARROW_HAVE_RUNTIME_AVX2)
   -            ,
   -            {DispatchLevel::AVX2, GreaterThanBitmapAvx2}
   +      , { DispatchLevel::AVX2, GreaterThanBitmapAvx2 }
    #endif
        };
      }
   @@ -57,10 +57,10 @@
      using FunctionType = decltype(&FindMinMax);
    
      static std::vector<std::pair<DispatchLevel, FunctionType>> implementations() {
   -    return {{DispatchLevel::NONE, standard::FindMinMaxImpl}
   +    return {
   +      { DispatchLevel::NONE, standard::FindMinMaxImpl }
    #if defined(ARROW_HAVE_RUNTIME_AVX2)
   -            ,
   -            {DispatchLevel::AVX2, FindMinMaxAvx2}
   +      , { DispatchLevel::AVX2, FindMinMaxAvx2 }
    #endif
        };
      }
   ```
   
   Could you try `archery lint --clang-format --fix` instead?
   https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977089562


##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));
+
+    // For this file, 3rd index value is null
+    if (i == 2) {
+      ASSERT_TRUE(is_null);
+    } else {
+      ASSERT_FALSE(is_null);
+    }
+  }
+
+  ASSERT_FALSE(scanner->HasNext());
+  ASSERT_FALSE(scanner->NextValue(&val, &is_null));
+}
+
+TEST_F(TestBooleanRLE, TestBatchRead) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto col = std::dynamic_pointer_cast<BoolReader>(group->Column(0));
+
+  // This file only has 8 rows
+  ASSERT_EQ(8, reader_->metadata()->num_rows());
+  // This file only has 1 row group
+  ASSERT_EQ(1, reader_->metadata()->num_row_groups());
+  // Size of the metadata is 106 bytes
+  ASSERT_EQ(106, reader_->metadata()->size());
+  // This row group must have 8 rows
+  ASSERT_EQ(8, group->metadata()->num_rows());
+
+  // Check if the column is encoded with RLE
+  auto col_chunk = group->metadata()->ColumnChunk(0);
+  ASSERT_TRUE(std::find(col_chunk->encodings().begin(), col_chunk->encodings().end(),
+                        Encoding::RLE) != col_chunk->encodings().end());
+
+  // Assert column has values to be read
+  ASSERT_TRUE(col->HasNext());
+  int64_t curr_batch_read = 0;
+
+  const int16_t batch_size = 8;
+  int16_t def_levels[batch_size];
+  int16_t rep_levels[batch_size];
+  bool values[batch_size - 1];
+
+  auto levels_read =
+      col->ReadBatch(batch_size, def_levels, rep_levels, values, &curr_batch_read);
+  ASSERT_EQ(batch_size, levels_read);
+
+  // Since one value is a null value, expect batches read to be one less than indicated
+  // batch_size
+  ASSERT_EQ(batch_size - 1, curr_batch_read);
+
+  // 3rd index is null value
+  ASSERT_THAT(def_levels, testing::ElementsAre(1, 1, 0, 1, 1, 1, 1, 1));
+
+  // Validate inserted data is as expected
+  ASSERT_THAT(values, testing::ElementsAre(1, 0, 1, 1, 0, 0, 1));

Review Comment:
   Hmmm, my bad. I thought the test case had both. 
   No, Boolean values use both` bit-packed-run` and `rle-run`. 
   
   I validated against another file : [rle_boolean_encoding.parquet](https://github.com/sfc-gh-nthimmegowda/parquet-testing/blob/master/data/rle_boolean_encoding.parquet). I could find both `is_literal` as true and false. 
   
   I beleive since the test file contains only some data (8 rows) it could not go into `<rle-run>` , however large amount of rows will move it. 
   My tentative plan is to replace the current test file with a new one with more rows. I would have done it in this PR, but the test files come from a submodule which require another PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] pitrou commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
pitrou commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1277369600

   This PR triggers undefined behavior in fuzzing builds:
   ```
   /build/build-fuzz/debug/parquet-arrow-fuzz: Running 1 inputs 1 time(s) each.
   Running: clusterfuzz-testcase-minimized-parquet-arrow-fuzz-5017972913864704
   /home/antoine/arrow/dev/cpp/src/arrow/util/bit_stream_utils.h:415:42: runtime error: load of value 32, which is not a valid value for type 'bool'
   SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/antoine/arrow/dev/cpp/src/arrow/util/bit_stream_utils.h:415:42 in 
   ==34310== ERROR: libFuzzer: deadly signal
       #0 0x55771d20cef1 in __sanitizer_print_stack_trace (/build/build-fuzz/debug/parquet-arrow-fuzz+0xe5ef1) (BuildId: 7423ef34b89be7542c884a94acdc1490cca072b4)
       #1 0x55771d17fe77 in fuzzer::PrintStackTrace() crtstuff.c
       #2 0x55771d1659a3 in fuzzer::Fuzzer::CrashCallback() crtstuff.c
       #3 0x7f9ccb2e741f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1441f) (BuildId: 7b4536f41cdaa5888408e82d0836e33dcf436466)
       #4 0x7f9ccb10900a in __libc_signal_restore_set /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/internal-signals.h:86:3
       #5 0x7f9ccb10900a in raise /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/raise.c:48:3
       #6 0x7f9ccb0e8858 in abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79:7
       #7 0x55771d227176 in __sanitizer::Abort() crtstuff.c
       #8 0x55771d225000 in __sanitizer::Die() crtstuff.c
       #9 0x55771d238d3b in __ubsan::ScopedReport::~ScopedReport() crtstuff.c
       #10 0x55771d23baea in handleLoadInvalidValue(__ubsan::InvalidValueData*, unsigned long, __ubsan::ReportOptions) crtstuff.c
       #11 0x55771d23bb2d in __ubsan_handle_load_invalid_value_abort (/build/build-fuzz/debug/parquet-arrow-fuzz+0x114b2d) (BuildId: 7423ef34b89be7542c884a94acdc1490cca072b4)
       #12 0x7f9ce5fbe8fa in bool arrow::bit_util::BitReader::GetAligned<bool>(int, bool*) /home/antoine/arrow/dev/cpp/src/arrow/util/bit_stream_utils.h:415:42
       #13 0x7f9ce5fbc533 in bool arrow::util::RleDecoder::NextCounts<bool>() /home/antoine/arrow/dev/cpp/src/arrow/util/rle_encoding.h:663:22
       #14 0x7f9ce5fb9087 in int arrow::util::RleDecoder::GetBatch<bool>(bool*, int) /home/antoine/arrow/dev/cpp/src/arrow/util/rle_encoding.h:329:12
       #15 0x7f9ce5faa13d in parquet::(anonymous namespace)::RleBooleanDecoder::Decode(bool*, int) /home/antoine/arrow/dev/cpp/src/parquet/encoding.cc:2388:19
       #16 0x7f9ce5d179a8 in parquet::internal::(anonymous namespace)::TypedRecordReader<parquet::PhysicalType<(parquet::Type::type)0> >::ReadValuesDense(long) /home/antoine/arrow/dev/cpp/src/parquet/column_reader.cc:1531:33
       #17 0x7f9ce5d1ac37 in parquet::internal::(anonymous namespace)::TypedRecordReader<parquet::PhysicalType<(parquet::Type::type)0> >::ReadRecordData(long) /home/antoine/arrow/dev/cpp/src/parquet/column_reader.cc:1575:7
       #18 0x7f9ce5d139b4 in parquet::internal::(anonymous namespace)::TypedRecordReader<parquet::PhysicalType<(parquet::Type::type)0> >::ReadRecords(long) /home/antoine/arrow/dev/cpp/src/parquet/column_reader.cc:1331:25
       #19 0x7f9ce572137d in parquet::arrow::(anonymous namespace)::LeafReader::LoadBatch(long) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:479:46
       #20 0x7f9ce571821e in parquet::arrow::ColumnReaderImpl::NextBatch(long, std::shared_ptr<arrow::ChunkedArray>*) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:109:5
       #21 0x7f9ce5785828 in parquet::arrow::(anonymous namespace)::FileReaderImpl::ReadColumn(int, std::vector<int, std::allocator<int> > const&, parquet::arrow::ColumnReader*, std::shared_ptr<arrow::ChunkedArray>*) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:285:20
       #22 0x7f9ce57fe70b in parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, arrow::internal::Executor*)::$_4::operator()(unsigned long, std::shared_ptr<parquet::arrow::ColumnReaderImpl>) const /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:1236:5
       #23 0x7f9ce57fa00c in arrow::Future<std::vector<std::shared_ptr<arrow::ChunkedArray>, std::allocator<std::shared_ptr<arrow::ChunkedArray> > > > arrow::internal::OptionalParallelForAsync<parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, arrow::internal::Executor*)::$_4&, std::shared_ptr<parquet::arrow::ColumnReaderImpl>, std::shared_ptr<arrow::ChunkedArray> >(bool, std::vector<std::shared_ptr<parquet::arrow::ColumnReaderImpl>, std::allocator<std::shared_ptr<parquet::arrow::ColumnReaderImpl> > >, parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, arrow::internal::Executor*)::$_4&, arrow::internal::Executor*) /hom
 e/antoine/arrow/dev/cpp/src/arrow/util/parallel.h:95:7
       #24 0x7f9ce57f89bb in parquet::arrow::(anonymous namespace)::FileReaderImpl::DecodeRowGroups(std::shared_ptr<parquet::arrow::(anonymous namespace)::FileReaderImpl>, std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, arrow::internal::Executor*) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:1254:10
       #25 0x7f9ce56f9266 in parquet::arrow::(anonymous namespace)::FileReaderImpl::ReadRowGroups(std::vector<int, std::allocator<int> > const&, std::vector<int, std::allocator<int> > const&, std::shared_ptr<arrow::Table>*) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:1215:14
       #26 0x7f9ce56f7e57 in parquet::arrow::(anonymous namespace)::FileReaderImpl::ReadRowGroup(int, std::vector<int, std::allocator<int> > const&, std::shared_ptr<arrow::Table>*) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:322:12
       #27 0x7f9ce56f83ab in parquet::arrow::(anonymous namespace)::FileReaderImpl::ReadRowGroup(int, std::shared_ptr<arrow::Table>*) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:326:12
       #28 0x7f9ce56e85d1 in parquet::arrow::internal::FuzzReader(std::unique_ptr<parquet::arrow::FileReader, std::default_delete<parquet::arrow::FileReader> >) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:1338:37
       #29 0x7f9ce56e9c35 in parquet::arrow::internal::FuzzReader(unsigned char const*, long) /home/antoine/arrow/dev/cpp/src/parquet/arrow/reader.cc:1355:10
       #30 0x55771d2402e7 in LLVMFuzzerTestOneInput /home/antoine/arrow/dev/cpp/src/parquet/arrow/fuzz.cc:22:17
       #31 0x55771d1670b3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) crtstuff.c
       #32 0x55771d15146f in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) crtstuff.c
       #33 0x55771d157176 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) crtstuff.c
       #34 0x55771d1807b2 in main (/build/build-fuzz/debug/parquet-arrow-fuzz+0x597b2) (BuildId: 7423ef34b89be7542c884a94acdc1490cca072b4)
       #35 0x7f9ccb0ea082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
       #36 0x55771d14bc5d in _start (/build/build-fuzz/debug/parquet-arrow-fuzz+0x24c5d) (BuildId: 7423ef34b89be7542c884a94acdc1490cca072b4)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1264586155

   > Could you confirm this failure? https://github.com/apache/arrow/actions/runs/3155990217/jobs/5136453754#step:6:3035
   
   Hmmm, this was weird. I identified where it was failing in ASAN and fixed it. Validated in my fork that all C++ gates pass as well. 
   https://github.com/sfc-gh-nthimmegowda/arrow/actions/runs/3167603155


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974954780


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   Yes, the test uses this method. 
   
   No, this does not write each value as 1 byte. This also write 1 bit value similarly to `PlainBooleanDecoder::Decode(uint8_t* buffer)`. The `PlainDecoder` calls `GetBatch` specifying bit-width as 1. 
   `bit_reader_->GetBatch(1, buffer, max_values)`. 
   
   RleBooleanDecoder also calls this internally in the same fashion.
   https://github.com/apache/arrow/blob/8daa7a4ed5629c0020dadf7325a6b523bdfc62e9/cpp/src/arrow/util/rle_encoding.h#L320
   Here bit-width is 1, as defined during initialization. 
   ```
       decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
                                                              /*bit_width=*/1);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r973666767


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");

Review Comment:
   Modified error message to show invalid length. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r973666794


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");
+      }
+      // Load the first 4 bytes, which indicates the legnth
+      num_bytes = ::arrow::util::SafeLoadAs<int32_t>(data);

Review Comment:
   Converted to unsignedint32_t and parsed as little endian. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on pull request #14147: ARROW-17450 : Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1248778935

   @kou Could you please review this change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974763276


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;

Review Comment:
   Changing to `bool` or `decoder_->GetBatch()` fails in compilation stage. 
   
   This barfs at compile time check with 
   ```
   bit_stream_utils.h:415:41: error: no matching function for call to ‘FromLittleEndian(bool&)’
     415 |   *v = arrow::bit_util::FromLittleEndian(*v);
   
   arrow/util/type_traits.h: In substitution of ‘template<class T, class ... Args> using EnableIfIsOneOf = typename std::enable_if<arrow::internal::IsOneOf<T, Args ...>::value, T>::type [with T = bool; Args = {long int, long unsigned int, int, unsigned int, short int, short unsigned int, unsigned char, signed char, float, double}]’:
   ```
   
   
   Code path 
   **RleDecoder::GetBatch(T*, int) -> RleDecoder::NextCounts() -> BitReader::GetAligned(int, T*)**
   
   In BitReader::GetAligned() :::: Line 415 , there is a call to bit_util::FromLittleEndian. 
   However FromLittleEndian can be called on only one of the following types int8...int64, float and double, whereas in this case it is bool All the accepted values are at the byte boundary / 8 bits --- 8,16,32. I am not exactly sure how boolean can fit into this structure. Little Endian or Big Endian wouldn't matter for boolean, but I worry this might change the computation because of the alignment of byte boundary. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974958686


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;

Review Comment:
   Parquet uses `0` and `1` for boolean in Parquet (Apart from nulls). 
   Size of boolean value in Parquet is 1 bit (Logically).
   
   Boolean is only supported in 2 encoding - [Plain](https://parquet.apache.org/docs/file-format/data-pages/encodings/#a-nameplainaplain-plain--0) and [RLE](https://parquet.apache.org/docs/file-format/data-pages/encodings/#a-namerlearun-length-encoding--bit-packing-hybrid-rle--3). Both of them use some part of bit-packing. Although in RLE cases, we might have lesser size because of continuous number, but physically never greater than 1 bit. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977099469


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   My bad again, I assumed `col->ReadBatch` , will internally call `decoder_->GetBatch`. 
   
   This method is covered in test `VectorBooleanTest`
   I went looking into the history why a boolean decoder has bits written to `uint8_t` datatype. This was introduced in this [commit](https://github.com/apache/arrow/commit/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc#diff-61373111f1c45b61c1232091a83f44e66895cf9d27b264d9cbfceac45ff87941). 
   
   This was introduced for testing purposes. The data is encoded with data from `vector<bool>` however it not possible to get reference for `bool` from `vector<bool>` which is needed to pass to Decode function
   `int Decode(bool* buffer, int max_values)
   `. [Link](https://en.wikipedia.org/w/index.php?title=Sequence_container_(C%2B%2B)&oldid=767869909#Specialization_for_bool) . 
   To bypass we introduced a reader with `uint_8` datatype. 
   
   I think a refactor on this is due, which is my next plan, that we replace `vector<bool>` with `bool[]` in test case and remove the class `BooleanDecoder`
   
   Proposed fix is here. Not including in this PR for compartmentalization. 
   
   ```
   diff --git a/cpp/src/parquet/encoding.cc b/cpp/src/parquet/encoding.cc
   index a862743d5..4a4646724 100644
   --- a/cpp/src/parquet/encoding.cc
   +++ b/cpp/src/parquet/encoding.cc
   @@ -1147,8 +1147,7 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
    }
    
    class PlainBooleanDecoder : public DecoderImpl,
   -                            virtual public TypedDecoder<BooleanType>,
   -                            virtual public BooleanDecoder {
   +                            virtual public TypedDecoder<BooleanType> {
     public:
      explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
      void SetData(int num_values, const uint8_t* data, int len) override;
   @@ -2358,7 +2357,7 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
    // ----------------------------------------------------------------------
    // RLE_BOOLEAN_DECODER
    
   -class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
   +class RleBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
     public:
      explicit RleBooleanDecoder(const ColumnDescriptor* descr)
          : DecoderImpl(descr, Encoding::RLE) {}
   diff --git a/cpp/src/parquet/encoding.h b/cpp/src/parquet/encoding.h
   index b9ca7a7ee..c32da9979 100644
   --- a/cpp/src/parquet/encoding.h
   +++ b/cpp/src/parquet/encoding.h
   @@ -65,7 +65,7 @@ using FLBAEncoder = TypedEncoder<FLBAType>;
    template <typename DType>
    class TypedDecoder;
    
   -class BooleanDecoder;
   +using BooleanDecoder = TypedDecoder<BooleanType>;
    using Int32Decoder = TypedDecoder<Int32Type>;
    using Int64Decoder = TypedDecoder<Int64Type>;
    using Int96Decoder = TypedDecoder<Int96Type>;
   @@ -394,12 +394,6 @@ class DictDecoder : virtual public TypedDecoder<DType> {
    // ----------------------------------------------------------------------
    // TypedEncoder specializations, traits, and factory functions
    
   -class BooleanDecoder : virtual public TypedDecoder<BooleanType> {
   - public:
   -  using TypedDecoder<BooleanType>::Decode;
   -  virtual int Decode(uint8_t* buffer, int max_values) = 0;
   -};
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977223220


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   OK. Do you want to work on it in this pull request or a follow-up pull request?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977086945


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;

Review Comment:
   Agreed. This is apt and concise. I have changed my implementation as recommended. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1263025916

   Could you rebase and resolve the current conflict?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r973666875


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2762,6 +2839,11 @@ std::unique_ptr<Decoder> MakeDecoder(Type::type type_num, Encoding::type encodin
       return std::unique_ptr<Decoder>(new DeltaLengthByteArrayDecoder(descr));
     }
     throw ParquetException("DELTA_LENGTH_BYTE_ARRAY only supports BYTE_ARRAY");
+  } else if (encoding == Encoding::RLE) {
+    if (type_num == Type::BOOLEAN) {
+      return std::unique_ptr<Decoder>(new RleBooleanDecoder(descr));
+    }
+    throw ParquetException("RLE encoding only supports BINARY");

Review Comment:
   Ooops. My bad. Changed to BOOLEAN. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974763965


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) {

Review Comment:
   Oops ! My bad . Added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974949651


##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));
+
+    // For this file, 3rd index value is null
+    if (i == 2) {
+      ASSERT_TRUE(is_null);
+    } else {
+      ASSERT_FALSE(is_null);
+    }
+  }
+
+  ASSERT_FALSE(scanner->HasNext());
+  ASSERT_FALSE(scanner->NextValue(&val, &is_null));
+}
+
+TEST_F(TestBooleanRLE, TestBatchRead) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto col = std::dynamic_pointer_cast<BoolReader>(group->Column(0));
+
+  // This file only has 8 rows
+  ASSERT_EQ(8, reader_->metadata()->num_rows());
+  // This file only has 1 row group
+  ASSERT_EQ(1, reader_->metadata()->num_row_groups());
+  // Size of the metadata is 106 bytes
+  ASSERT_EQ(106, reader_->metadata()->size());
+  // This row group must have 8 rows
+  ASSERT_EQ(8, group->metadata()->num_rows());
+
+  // Check if the column is encoded with RLE
+  auto col_chunk = group->metadata()->ColumnChunk(0);
+  ASSERT_TRUE(std::find(col_chunk->encodings().begin(), col_chunk->encodings().end(),
+                        Encoding::RLE) != col_chunk->encodings().end());
+
+  // Assert column has values to be read
+  ASSERT_TRUE(col->HasNext());
+  int64_t curr_batch_read = 0;
+
+  const int16_t batch_size = 8;
+  int16_t def_levels[batch_size];
+  int16_t rep_levels[batch_size];
+  bool values[batch_size - 1];
+
+  auto levels_read =
+      col->ReadBatch(batch_size, def_levels, rep_levels, values, &curr_batch_read);
+  ASSERT_EQ(batch_size, levels_read);
+
+  // Since one value is a null value, expect batches read to be one less than indicated
+  // batch_size
+  ASSERT_EQ(batch_size - 1, curr_batch_read);
+
+  // 3rd index is null value
+  ASSERT_THAT(def_levels, testing::ElementsAre(1, 1, 0, 1, 1, 1, 1, 1));
+
+  // Validate inserted data is as expected
+  ASSERT_THAT(values, testing::ElementsAre(1, 0, 1, 1, 0, 0, 1));

Review Comment:
   Yes, the input has both `<bit-packed-run>` and `<rle-run>`. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1250201672

   > Could you fix style by `cmake --build BUILD_DIR --config Debug --target format`?
   
   Ack. Ran `--target format` . Had changes in 2 other files which I included since those were minor. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r984152370


##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));
+
+    // For this file, 3rd index value is null
+    if (i == 2) {
+      ASSERT_TRUE(is_null);
+    } else {
+      ASSERT_FALSE(is_null);
+    }
+  }
+
+  ASSERT_FALSE(scanner->HasNext());
+  ASSERT_FALSE(scanner->NextValue(&val, &is_null));
+}
+
+TEST_F(TestBooleanRLE, TestBatchRead) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto col = std::dynamic_pointer_cast<BoolReader>(group->Column(0));
+
+  // This file only has 8 rows
+  ASSERT_EQ(8, reader_->metadata()->num_rows());
+  // This file only has 1 row group
+  ASSERT_EQ(1, reader_->metadata()->num_row_groups());
+  // Size of the metadata is 106 bytes
+  ASSERT_EQ(106, reader_->metadata()->size());
+  // This row group must have 8 rows
+  ASSERT_EQ(8, group->metadata()->num_rows());
+
+  // Check if the column is encoded with RLE
+  auto col_chunk = group->metadata()->ColumnChunk(0);
+  ASSERT_TRUE(std::find(col_chunk->encodings().begin(), col_chunk->encodings().end(),
+                        Encoding::RLE) != col_chunk->encodings().end());
+
+  // Assert column has values to be read
+  ASSERT_TRUE(col->HasNext());
+  int64_t curr_batch_read = 0;
+
+  const int16_t batch_size = 8;
+  int16_t def_levels[batch_size];
+  int16_t rep_levels[batch_size];
+  bool values[batch_size - 1];
+
+  auto levels_read =
+      col->ReadBatch(batch_size, def_levels, rep_levels, values, &curr_batch_read);
+  ASSERT_EQ(batch_size, levels_read);
+
+  // Since one value is a null value, expect batches read to be one less than indicated
+  // batch_size
+  ASSERT_EQ(batch_size - 1, curr_batch_read);
+
+  // 3rd index is null value
+  ASSERT_THAT(def_levels, testing::ElementsAre(1, 1, 0, 1, 1, 1, 1, 1));
+
+  // Validate inserted data is as expected
+  ASSERT_THAT(values, testing::ElementsAre(1, 0, 1, 1, 0, 0, 1));

Review Comment:
   Ack. Updated test file file in [parquet-testing](https://github.com/apache/parquet-testing) and updating in this pull request. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1263024335

   Hi @kou , 
   Updated the new test file and refactored the changes. Please have a look. 
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r988452821


##########
cpp/src/parquet/encoding.cc:
##########
@@ -1146,15 +1146,11 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
   return max_values;
 }
 
-class PlainBooleanDecoder : public DecoderImpl,
-                            virtual public TypedDecoder<BooleanType>,
-                            virtual public BooleanDecoder {
+class PlainBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
  public:
   explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
   void SetData(int num_values, const uint8_t* data, int len) override;
 
-  // Two flavors of bool decoding
-  int Decode(uint8_t* buffer, int max_values) override;

Review Comment:
   @kou I believe we can. Setting out a PR later today for this. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #14147: ARROW-17450 : Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1248637407

   https://issues.apache.org/jira/browse/ARROW-17450


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974763886


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) {
+    max_values = std::min(max_values, num_values_);
+    if (!decoder_->GetBatch(buffer, max_values)) {
+      ParquetException::EofException();
+    }
+
+    num_values_ -= max_values;

Review Comment:
   Agreed. Changed to validate that number of read values is equal to `max_values` . This makes it inline with rest of GetBatch()



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r988379907


##########
cpp/src/parquet/encoding.cc:
##########
@@ -1146,15 +1146,11 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
   return max_values;
 }
 
-class PlainBooleanDecoder : public DecoderImpl,
-                            virtual public TypedDecoder<BooleanType>,
-                            virtual public BooleanDecoder {
+class PlainBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
  public:
   explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
   void SetData(int num_values, const uint8_t* data, int len) override;
 
-  // Two flavors of bool decoding
-  int Decode(uint8_t* buffer, int max_values) override;

Review Comment:
   See https://github.com/apache/arrow/pull/14147#discussion_r977099469 .
   Could you share your downstream code that uses this?
   
   @sfc-gh-nthimmegowda Can we keep backward compatibility for this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] wgtmac commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
wgtmac commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r990164853


##########
cpp/src/parquet/encoding.cc:
##########
@@ -1146,15 +1146,11 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
   return max_values;
 }
 
-class PlainBooleanDecoder : public DecoderImpl,
-                            virtual public TypedDecoder<BooleanType>,
-                            virtual public BooleanDecoder {
+class PlainBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
  public:
   explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
   void SetData(int num_values, const uint8_t* data, int len) override;
 
-  // Two flavors of bool decoding
-  int Decode(uint8_t* buffer, int max_values) override;

Review Comment:
   I have used the template TypedDecoder class which is created by MakeTypedDecoder function defined here: https://github.com/apache/arrow/blob/master/cpp/src/parquet/encoding.h#L447. Then it is casted to BooleanDecoder to use the uint8_t variant. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] wgtmac commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
wgtmac commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r989044314


##########
cpp/src/parquet/encoding.cc:
##########
@@ -1146,15 +1146,11 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
   return max_values;
 }
 
-class PlainBooleanDecoder : public DecoderImpl,
-                            virtual public TypedDecoder<BooleanType>,
-                            virtual public BooleanDecoder {
+class PlainBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
  public:
   explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
   void SetData(int num_values, const uint8_t* data, int len) override;
 
-  // Two flavors of bool decoding
-  int Decode(uint8_t* buffer, int max_values) override;

Review Comment:
   > See [#14147 (comment)](https://github.com/apache/arrow/pull/14147#discussion_r977099469) . Could you share your downstream code that uses this?
   > 
   > @sfc-gh-nthimmegowda Can we keep backward compatibility for this?
   
   It pretty straight-forward. The downstream code can use a vector of uint8_t instead of bool to hold a vector of decoded boolean values.
   
   @sfc-gh-nthimmegowda Thanks for keeping backward compatibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977099469


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   My bad again, I assumed `col->ReadBatch` , will internally call `decoder_->GetBatch`. 
   
   This method is covered in test `VectorBooleanTest`
   I went looking into the history why a boolean decoder has bits written to `uint8_t` datatype. This was introduced in this [commit](https://github.com/apache/arrow/commit/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc#diff-61373111f1c45b61c1232091a83f44e66895cf9d27b264d9cbfceac45ff87941). 
   
   This was introduced for testing purposes. The data is encoded with data from `vector<bool>` however it not possible to get reference for `bool` from `vector<bool>` which is needed to pass to Decode function
   `int Decode(bool* buffer, int max_values)
   `. [Link](https://en.wikipedia.org/w/index.php?title=Sequence_container_(C%2B%2B)&oldid=767869909#Specialization_for_bool) . 
   To bypass we introduced a reader with `uint_8` datatype. 
   
   I think a refactor on this is due, which is my next plan, that we replace `vector<bool>` with `bool[]` in test case and remove the class `BooleanDecoder`
   
   Proposed fix is here. Not including in this PR for compartmentalization. 
   
   ```diff
   diff --git a/cpp/src/parquet/encoding.cc b/cpp/src/parquet/encoding.cc
   index a862743d5..4a4646724 100644
   --- a/cpp/src/parquet/encoding.cc
   +++ b/cpp/src/parquet/encoding.cc
   @@ -1147,8 +1147,7 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
    }
    
    class PlainBooleanDecoder : public DecoderImpl,
   -                            virtual public TypedDecoder<BooleanType>,
   -                            virtual public BooleanDecoder {
   +                            virtual public TypedDecoder<BooleanType> {
     public:
      explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
      void SetData(int num_values, const uint8_t* data, int len) override;
   @@ -2358,7 +2357,7 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
    // ----------------------------------------------------------------------
    // RLE_BOOLEAN_DECODER
    
   -class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
   +class RleBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
     public:
      explicit RleBooleanDecoder(const ColumnDescriptor* descr)
          : DecoderImpl(descr, Encoding::RLE) {}
   diff --git a/cpp/src/parquet/encoding.h b/cpp/src/parquet/encoding.h
   index b9ca7a7ee..c32da9979 100644
   --- a/cpp/src/parquet/encoding.h
   +++ b/cpp/src/parquet/encoding.h
   @@ -65,7 +65,7 @@ using FLBAEncoder = TypedEncoder<FLBAType>;
    template <typename DType>
    class TypedDecoder;
    
   -class BooleanDecoder;
   +using BooleanDecoder = TypedDecoder<BooleanType>;
    using Int32Decoder = TypedDecoder<Int32Type>;
    using Int64Decoder = TypedDecoder<Int64Type>;
    using Int96Decoder = TypedDecoder<Int96Type>;
   @@ -394,12 +394,6 @@ class DictDecoder : virtual public TypedDecoder<DType> {
    // ----------------------------------------------------------------------
    // TypedEncoder specializations, traits, and factory functions
    
   -class BooleanDecoder : virtual public TypedDecoder<BooleanType> {
   - public:
   -  using TypedDecoder<BooleanType>::Decode;
   -  virtual int Decode(uint8_t* buffer, int max_values) = 0;
   -};
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1255495820

   Could you open a new Jira issue for it? I think that it's better that we work on it in a separated pull request.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r972757477


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");

Review Comment:
   How about showing the invalid length too?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");
+      }
+      // Load the first 4 bytes, which indicates the legnth
+      num_bytes = ::arrow::util::SafeLoadAs<int32_t>(data);
+      if (num_bytes < 0 || num_bytes > len - 4) {
+        throw ParquetException("Received invalid number of bytes (corrupt data page?)");

Review Comment:
   Could you also show the invalid value?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");
+      }
+      // Load the first 4 bytes, which indicates the legnth
+      num_bytes = ::arrow::util::SafeLoadAs<int32_t>(data);
+      if (num_bytes < 0 || num_bytes > len - 4) {
+        throw ParquetException("Received invalid number of bytes (corrupt data page?)");
+      }
+
+      const uint8_t* decoder_data = data + 4;
+      if (len == 0) {

Review Comment:
   It seems that `len` is always `>= 4` here.



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2762,6 +2839,11 @@ std::unique_ptr<Decoder> MakeDecoder(Type::type type_num, Encoding::type encodin
       return std::unique_ptr<Decoder>(new DeltaLengthByteArrayDecoder(descr));
     }
     throw ParquetException("DELTA_LENGTH_BYTE_ARRAY only supports BYTE_ARRAY");
+  } else if (encoding == Encoding::RLE) {
+    if (type_num == Type::BOOLEAN) {
+      return std::unique_ptr<Decoder>(new RleBooleanDecoder(descr));
+    }
+    throw ParquetException("RLE encoding only supports BINARY");

Review Comment:
   boolean?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");
+      }
+      // Load the first 4 bytes, which indicates the legnth
+      num_bytes = ::arrow::util::SafeLoadAs<int32_t>(data);

Review Comment:
   `uint32_t`?
   
   https://parquet.apache.org/docs/file-format/data-pages/encodings/#a-namerlearun-length-encoding--bit-packing-hybrid-rle--3
   
   > length := length of the <encoded-data> in bytes stored as 4 bytes little endian (unsigned int32)
   
   Could you parse this as little endian value on both little/big endian environment?
   
   We may be able to use `arrow/util/endian.h` for it...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974947517


##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));

Review Comment:
   Added. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974764215


##########
cpp/src/parquet/level_comparison.cc:
##########
@@ -44,10 +44,10 @@ struct GreaterThanDynamicFunction {
   using FunctionType = decltype(&GreaterThanBitmap);
 
   static std::vector<std::pair<DispatchLevel, FunctionType>> implementations() {
-    return {
-      { DispatchLevel::NONE, standard::GreaterThanBitmapImpl }
+    return {{DispatchLevel::NONE, standard::GreaterThanBitmapImpl}
 #if defined(ARROW_HAVE_RUNTIME_AVX2)
-      , { DispatchLevel::AVX2, GreaterThanBitmapAvx2 }
+            ,
+            {DispatchLevel::AVX2, GreaterThanBitmapAvx2}

Review Comment:
   You are right. I was using clang-format-14. Changed to use clang-format-12 and this gets it inline. 
   Waiting for Merge gates validation. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974763886


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) {
+    max_values = std::min(max_values, num_values_);
+    if (!decoder_->GetBatch(buffer, max_values)) {
+      ParquetException::EofException();
+    }
+
+    num_values_ -= max_values;

Review Comment:
   Agreed. Changed to validate that number of read values is equal to `max_values` . This makes it inline with rest of `GetBatch() `calls in the file. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r989374908


##########
cpp/src/parquet/encoding.cc:
##########
@@ -1146,15 +1146,11 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
   return max_values;
 }
 
-class PlainBooleanDecoder : public DecoderImpl,
-                            virtual public TypedDecoder<BooleanType>,
-                            virtual public BooleanDecoder {
+class PlainBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
  public:
   explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
   void SetData(int num_values, const uint8_t* data, int len) override;
 
-  // Two flavors of bool decoding
-  int Decode(uint8_t* buffer, int max_values) override;

Review Comment:
   @wgtmac 
   Can you share the code pointer , abstract is fine. 
   Because this method / function was an abstract function which was overriden, just wondering if you called in base class or called directly via `PlainBooleanEncoder` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r973666835


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");
+      }
+      // Load the first 4 bytes, which indicates the legnth
+      num_bytes = ::arrow::util::SafeLoadAs<int32_t>(data);
+      if (num_bytes < 0 || num_bytes > len - 4) {
+        throw ParquetException("Received invalid number of bytes (corrupt data page?)");
+      }
+
+      const uint8_t* decoder_data = data + 4;
+      if (len == 0) {

Review Comment:
   Agreed. This check is unnecessary. Removed. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r974887734


##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));
+
+    // For this file, 3rd index value is null
+    if (i == 2) {
+      ASSERT_TRUE(is_null);
+    } else {
+      ASSERT_FALSE(is_null);
+    }
+  }
+
+  ASSERT_FALSE(scanner->HasNext());
+  ASSERT_FALSE(scanner->NextValue(&val, &is_null));
+}
+
+TEST_F(TestBooleanRLE, TestBatchRead) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto col = std::dynamic_pointer_cast<BoolReader>(group->Column(0));
+
+  // This file only has 8 rows
+  ASSERT_EQ(8, reader_->metadata()->num_rows());
+  // This file only has 1 row group
+  ASSERT_EQ(1, reader_->metadata()->num_row_groups());
+  // Size of the metadata is 106 bytes
+  ASSERT_EQ(106, reader_->metadata()->size());
+  // This row group must have 8 rows
+  ASSERT_EQ(8, group->metadata()->num_rows());
+
+  // Check if the column is encoded with RLE
+  auto col_chunk = group->metadata()->ColumnChunk(0);
+  ASSERT_TRUE(std::find(col_chunk->encodings().begin(), col_chunk->encodings().end(),
+                        Encoding::RLE) != col_chunk->encodings().end());
+
+  // Assert column has values to be read
+  ASSERT_TRUE(col->HasNext());
+  int64_t curr_batch_read = 0;
+
+  const int16_t batch_size = 8;
+  int16_t def_levels[batch_size];
+  int16_t rep_levels[batch_size];
+  bool values[batch_size - 1];
+
+  auto levels_read =
+      col->ReadBatch(batch_size, def_levels, rep_levels, values, &curr_batch_read);
+  ASSERT_EQ(batch_size, levels_read);
+
+  // Since one value is a null value, expect batches read to be one less than indicated
+  // batch_size
+  ASSERT_EQ(batch_size - 1, curr_batch_read);
+
+  // 3rd index is null value
+  ASSERT_THAT(def_levels, testing::ElementsAre(1, 1, 0, 1, 1, 1, 1, 1));
+
+  // Validate inserted data is as expected
+  ASSERT_THAT(values, testing::ElementsAre(1, 0, 1, 1, 0, 0, 1));

Review Comment:
   Does the input have both `<bit-packed-run>` and `<rle-run>` cases in https://parquet.apache.org/docs/file-format/data-pages/encodings/#a-namerlearun-length-encoding--bit-packing-hybrid-rle--3 ?



##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));

Review Comment:
   Could you also validate `val`?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   It seems that this writes each value as 1byte value to `buffer`.
   But `PlainBooleanDecoder::Decode(uint8_t* buffer)` writes each value as 1bit value to `buffer`.
   
   Does the added tests use this method?



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;

Review Comment:
   Hmm. Do you know what value is used for boolean in Parquet? (`0` and `1`?) And do you also know the size of a boolean value in Parquet? (1bit? 8bits?)
   I can't find them in https://parquet.apache.org/docs/file-format/data-pages/encodings/ .
   
   I think that we need to use the same type for this. If Parquet uses 8bits, we can use `int8_t` or `uint8_t` here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r975997169


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   It seems that `bit_reader_->GetBatch(1, buffer, max_values)` with `T=uint8_t` read 1 bit data and write 8 bits data to `buffer`.
   
   See also:
   
   * https://github.com/apache/arrow/blob/8daa7a4ed5629c0020dadf7325a6b523bdfc62e9/cpp/src/arrow/util/bit_stream_utils.h#L337
   * https://github.com/apache/arrow/blob/8daa7a4ed5629c0020dadf7325a6b523bdfc62e9/cpp/src/arrow/util/bit_stream_utils.h#L279
   
   I wanted to confirm this behavior by debugger but it seems that this method isn't used in tests...



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,80 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > (uint32_t)(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    const uint8_t* decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;

Review Comment:
   Thanks.
   
   I re-read the implementation and understand that `RleDecoder` reads 1 bit in this case and write the read value to `buffer` with cast. (1 bit boolean value -> `static_cast<int>(1)` for `true`/`static_cast<int>(0)` for `false` in this case.) So we can use any type for `val`. But I think that `bool` is better than `int` because we want to return `bool` values.
   
   How about this?
   
   ```diff
   diff --git a/cpp/src/arrow/util/endian.h b/cpp/src/arrow/util/endian.h
   index f77077f809..d428287882 100644
   --- a/cpp/src/arrow/util/endian.h
   +++ b/cpp/src/arrow/util/endian.h
   @@ -122,28 +122,28 @@ static inline void ByteSwap(void* dst, const void* src, int len) {
    #if ARROW_LITTLE_ENDIAN
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T ToBigEndian(T value) {
      return ByteSwap(value);
    }
    
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T ToLittleEndian(T value) {
      return value;
    }
    #else
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T ToBigEndian(T value) {
      return value;
    }
    
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T ToLittleEndian(T value) {
      return ByteSwap(value);
    }
   @@ -153,28 +153,28 @@ static inline T ToLittleEndian(T value) {
    #if ARROW_LITTLE_ENDIAN
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T FromBigEndian(T value) {
      return ByteSwap(value);
    }
    
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T FromLittleEndian(T value) {
      return value;
    }
    #else
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T FromBigEndian(T value) {
      return value;
    }
    
    template <typename T, typename = internal::EnableIfIsOneOf<
                              T, int64_t, uint64_t, int32_t, uint32_t, int16_t, uint16_t,
   -                          uint8_t, int8_t, float, double>>
   +                          uint8_t, int8_t, float, double, bool>>
    static inline T FromLittleEndian(T value) {
      return ByteSwap(value);
    }
   diff --git a/cpp/src/parquet/encoding.cc b/cpp/src/parquet/encoding.cc
   index 98626c9791..b9257aab22 100644
   --- a/cpp/src/parquet/encoding.cc
   +++ b/cpp/src/parquet/encoding.cc
   @@ -2386,18 +2386,8 @@ class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
    
      int Decode(bool* buffer, int max_values) override {
        max_values = std::min(max_values, num_values_);
   -    int val = 0;
   -
   -    for (int i = 0; i < max_values; ++i) {
   -      if (!decoder_->Get(&val)) {
   -        throw ParquetException("Unable to parse bits for position (0 based) : " +
   -                               std::to_string(i) + " (corrupt data page?)");
   -      }
   -      if (val) {
   -        buffer[i] = true;
   -      } else {
   -        buffer[i] = false;
   -      }
   +    if (decoder_->GetBatch(buffer, max_values) != max_values) {
   +      ParquetException::EofException();
        }
        num_values_ -= max_values;
        return max_values;
   ```



##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   I applied the following patch
   
   ```diff
   diff --git a/cpp/src/parquet/encoding.cc b/cpp/src/parquet/encoding.cc
   index 98626c9791..f3e498ac9d 100644
   --- a/cpp/src/parquet/encoding.cc
   +++ b/cpp/src/parquet/encoding.cc
   @@ -2404,6 +2404,7 @@ class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
      }
    
      int Decode(uint8_t* buffer, int max_values) override {
   +    throw ParquetException("XXX");
        max_values = std::min(max_values, num_values_);
        if (decoder_->GetBatch(buffer, max_values) != max_values) {
          ParquetException::EofException();
   ```
   
   and ran `debug/parquet-reader-test` but no exception was raised:
   
   <details>
   
   ```console
   $ debug/parquet-reader-test
   Running main() from ./googletest/src/gtest_main.cc
   [==========] Running 82 tests from 27 test suites.
   [----------] Global test environment set-up.
   [----------] 10 tests from TestPrimitiveReader
   [ RUN      ] TestPrimitiveReader.TestInt32FlatRequired
   [       OK ] TestPrimitiveReader.TestInt32FlatRequired (22 ms)
   [ RUN      ] TestPrimitiveReader.TestInt32FlatOptional
   [       OK ] TestPrimitiveReader.TestInt32FlatOptional (5 ms)
   [ RUN      ] TestPrimitiveReader.TestInt32FlatRepeated
   [       OK ] TestPrimitiveReader.TestInt32FlatRepeated (6 ms)
   [ RUN      ] TestPrimitiveReader.TestInt32FlatRequiredSkip
   [       OK ] TestPrimitiveReader.TestInt32FlatRequiredSkip (0 ms)
   [ RUN      ] TestPrimitiveReader.TestReadValuesMissing
   [       OK ] TestPrimitiveReader.TestReadValuesMissing (0 ms)
   [ RUN      ] TestPrimitiveReader.TestRepetitionLvlBytesWithMaxRepetitionZero
   [       OK ] TestPrimitiveReader.TestRepetitionLvlBytesWithMaxRepetitionZero (0 ms)
   [ RUN      ] TestPrimitiveReader.TestReadValuesMissingWithDictionary
   [       OK ] TestPrimitiveReader.TestReadValuesMissingWithDictionary (0 ms)
   [ RUN      ] TestPrimitiveReader.TestDictionaryEncodedPages
   [       OK ] TestPrimitiveReader.TestDictionaryEncodedPages (0 ms)
   [ RUN      ] TestPrimitiveReader.TestDictionaryEncodedPagesWithExposeEncoding
   [       OK ] TestPrimitiveReader.TestDictionaryEncodedPagesWithExposeEncoding (0 ms)
   [ RUN      ] TestPrimitiveReader.TestNonDictionaryEncodedPagesWithExposeEncoding
   [       OK ] TestPrimitiveReader.TestNonDictionaryEncodedPagesWithExposeEncoding (0 ms)
   [----------] 10 tests from TestPrimitiveReader (35 ms total)
   
   [----------] 2 tests from TestColumnReader
   [ RUN      ] TestColumnReader.DefLevelsToBitmap
   [       OK ] TestColumnReader.DefLevelsToBitmap (0 ms)
   [ RUN      ] TestColumnReader.DefLevelsToBitmapPowerOfTwo
   [       OK ] TestColumnReader.DefLevelsToBitmapPowerOfTwo (0 ms)
   [----------] 2 tests from TestColumnReader (0 ms total)
   
   [----------] 1 test from GreaterThanBitmap
   [ RUN      ] GreaterThanBitmap.GeneratesExpectedBitmasks
   [       OK ] GreaterThanBitmap.GeneratesExpectedBitmasks (0 ms)
   [----------] 1 test from GreaterThanBitmap (0 ms total)
   
   [----------] 1 test from DefLevelsToBitmap
   [ RUN      ] DefLevelsToBitmap.WithRepetitionLevelFiltersOutEmptyListValues
   [       OK ] DefLevelsToBitmap.WithRepetitionLevelFiltersOutEmptyListValues (0 ms)
   [----------] 1 test from DefLevelsToBitmap (0 ms total)
   
   [----------] 5 tests from NestedListTest/0, where TypeParam = parquet::internal::RepDefLevelConverter<int>
   [ RUN      ] NestedListTest/0.OuterMostTest
   [       OK ] NestedListTest/0.OuterMostTest (0 ms)
   [ RUN      ] NestedListTest/0.MiddleListTest
   [       OK ] NestedListTest/0.MiddleListTest (0 ms)
   [ RUN      ] NestedListTest/0.InnerMostListTest
   [       OK ] NestedListTest/0.InnerMostListTest (0 ms)
   [ RUN      ] NestedListTest/0.SimpleLongList
   [       OK ] NestedListTest/0.SimpleLongList (0 ms)
   [ RUN      ] NestedListTest/0.TestOverflow
   [       OK ] NestedListTest/0.TestOverflow (0 ms)
   [----------] 5 tests from NestedListTest/0 (0 ms total)
   
   [----------] 5 tests from NestedListTest/1, where TypeParam = parquet::internal::RepDefLevelConverter<long>
   [ RUN      ] NestedListTest/1.OuterMostTest
   [       OK ] NestedListTest/1.OuterMostTest (0 ms)
   [ RUN      ] NestedListTest/1.MiddleListTest
   [       OK ] NestedListTest/1.MiddleListTest (0 ms)
   [ RUN      ] NestedListTest/1.InnerMostListTest
   [       OK ] NestedListTest/1.InnerMostListTest (0 ms)
   [ RUN      ] NestedListTest/1.SimpleLongList
   [       OK ] NestedListTest/1.SimpleLongList (0 ms)
   [ RUN      ] NestedListTest/1.TestOverflow
   [       OK ] NestedListTest/1.TestOverflow (0 ms)
   [----------] 5 tests from NestedListTest/1 (0 ms total)
   
   [----------] 1 test from TestOnlyExtractBitsSoftware
   [ RUN      ] TestOnlyExtractBitsSoftware.BasicTest
   [       OK ] TestOnlyExtractBitsSoftware.BasicTest (0 ms)
   [----------] 1 test from TestOnlyExtractBitsSoftware (0 ms total)
   
   [----------] 2 tests from TestFlatScanner/0, where TypeParam = parquet::PhysicalType<(parquet::Type::type)1>
   [ RUN      ] TestFlatScanner/0.TestPlainScanner
   [       OK ] TestFlatScanner/0.TestPlainScanner (2 ms)
   [ RUN      ] TestFlatScanner/0.TestDictScanner
   [       OK ] TestFlatScanner/0.TestDictScanner (2 ms)
   [----------] 2 tests from TestFlatScanner/0 (4 ms total)
   
   [----------] 2 tests from TestFlatScanner/1, where TypeParam = parquet::PhysicalType<(parquet::Type::type)2>
   [ RUN      ] TestFlatScanner/1.TestPlainScanner
   [       OK ] TestFlatScanner/1.TestPlainScanner (2 ms)
   [ RUN      ] TestFlatScanner/1.TestDictScanner
   [       OK ] TestFlatScanner/1.TestDictScanner (2 ms)
   [----------] 2 tests from TestFlatScanner/1 (4 ms total)
   
   [----------] 2 tests from TestFlatScanner/2, where TypeParam = parquet::PhysicalType<(parquet::Type::type)3>
   [ RUN      ] TestFlatScanner/2.TestPlainScanner
   [       OK ] TestFlatScanner/2.TestPlainScanner (2 ms)
   [ RUN      ] TestFlatScanner/2.TestDictScanner
   [       OK ] TestFlatScanner/2.TestDictScanner (2 ms)
   [----------] 2 tests from TestFlatScanner/2 (5 ms total)
   
   [----------] 2 tests from TestFlatScanner/3, where TypeParam = parquet::PhysicalType<(parquet::Type::type)4>
   [ RUN      ] TestFlatScanner/3.TestPlainScanner
   [       OK ] TestFlatScanner/3.TestPlainScanner (2 ms)
   [ RUN      ] TestFlatScanner/3.TestDictScanner
   [       OK ] TestFlatScanner/3.TestDictScanner (2 ms)
   [----------] 2 tests from TestFlatScanner/3 (4 ms total)
   
   [----------] 2 tests from TestFlatScanner/4, where TypeParam = parquet::PhysicalType<(parquet::Type::type)5>
   [ RUN      ] TestFlatScanner/4.TestPlainScanner
   [       OK ] TestFlatScanner/4.TestPlainScanner (2 ms)
   [ RUN      ] TestFlatScanner/4.TestDictScanner
   [       OK ] TestFlatScanner/4.TestDictScanner (2 ms)
   [----------] 2 tests from TestFlatScanner/4 (4 ms total)
   
   [----------] 2 tests from TestFlatScanner/5, where TypeParam = parquet::PhysicalType<(parquet::Type::type)6>
   [ RUN      ] TestFlatScanner/5.TestPlainScanner
   [       OK ] TestFlatScanner/5.TestPlainScanner (2 ms)
   [ RUN      ] TestFlatScanner/5.TestDictScanner
   [       OK ] TestFlatScanner/5.TestDictScanner (2 ms)
   [----------] 2 tests from TestFlatScanner/5 (5 ms total)
   
   [----------] 1 test from TestBooleanFlatScanner
   [ RUN      ] TestBooleanFlatScanner.TestPlainScanner
   [       OK ] TestBooleanFlatScanner.TestPlainScanner (2 ms)
   [----------] 1 test from TestBooleanFlatScanner (2 ms total)
   
   [----------] 6 tests from TestFLBAFlatScanner
   [ RUN      ] TestFLBAFlatScanner.TestPlainScanner
   [       OK ] TestFLBAFlatScanner.TestPlainScanner (2 ms)
   [ RUN      ] TestFLBAFlatScanner.TestDictScanner
   [       OK ] TestFLBAFlatScanner.TestDictScanner (2 ms)
   [ RUN      ] TestFLBAFlatScanner.TestPlainDictScanner
   [       OK ] TestFLBAFlatScanner.TestPlainDictScanner (2 ms)
   [ RUN      ] TestFLBAFlatScanner.TestSmallBatch
   [       OK ] TestFLBAFlatScanner.TestSmallBatch (0 ms)
   [ RUN      ] TestFLBAFlatScanner.TestDescriptorAPI
   [       OK ] TestFLBAFlatScanner.TestDescriptorAPI (0 ms)
   [ RUN      ] TestFLBAFlatScanner.TestFLBAPrinterNext
   [       OK ] TestFLBAFlatScanner.TestFLBAPrinterNext (0 ms)
   [----------] 6 tests from TestFLBAFlatScanner (9 ms total)
   
   [----------] 2 tests from TestBooleanRLE
   [ RUN      ] TestBooleanRLE.TestBooleanScanner
   [       OK ] TestBooleanRLE.TestBooleanScanner (0 ms)
   [ RUN      ] TestBooleanRLE.TestBatchRead
   [       OK ] TestBooleanRLE.TestBatchRead (0 ms)
   [----------] 2 tests from TestBooleanRLE (1 ms total)
   
   [----------] 2 tests from TestTextDeltaLengthByteArray
   [ RUN      ] TestTextDeltaLengthByteArray.TestTextScanner
   [       OK ] TestTextDeltaLengthByteArray.TestTextScanner (0 ms)
   [ RUN      ] TestTextDeltaLengthByteArray.TestBatchRead
   [       OK ] TestTextDeltaLengthByteArray.TestBatchRead (0 ms)
   [----------] 2 tests from TestTextDeltaLengthByteArray (1 ms total)
   
   [----------] 9 tests from TestAllTypesPlain
   [ RUN      ] TestAllTypesPlain.NoopConstructDestruct
   [       OK ] TestAllTypesPlain.NoopConstructDestruct (0 ms)
   [ RUN      ] TestAllTypesPlain.RowGroupMetaData
   [       OK ] TestAllTypesPlain.RowGroupMetaData (0 ms)
   [ RUN      ] TestAllTypesPlain.TestBatchRead
   [       OK ] TestAllTypesPlain.TestBatchRead (0 ms)
   [ RUN      ] TestAllTypesPlain.RowGroupColumnBoundchecking
   [       OK ] TestAllTypesPlain.RowGroupColumnBoundchecking (0 ms)
   [ RUN      ] TestAllTypesPlain.TestFlatScannerInt32
   [       OK ] TestAllTypesPlain.TestFlatScannerInt32 (0 ms)
   [ RUN      ] TestAllTypesPlain.TestSetScannerBatchSize
   [       OK ] TestAllTypesPlain.TestSetScannerBatchSize (0 ms)
   [ RUN      ] TestAllTypesPlain.DebugPrintWorks
   [       OK ] TestAllTypesPlain.DebugPrintWorks (0 ms)
   [ RUN      ] TestAllTypesPlain.ColumnSelection
   [       OK ] TestAllTypesPlain.ColumnSelection (0 ms)
   [ RUN      ] TestAllTypesPlain.ColumnSelectionOutOfRange
   [       OK ] TestAllTypesPlain.ColumnSelectionOutOfRange (0 ms)
   [----------] 9 tests from TestAllTypesPlain (2 ms total)
   
   [----------] 1 test from TestLocalFile
   [ RUN      ] TestLocalFile.OpenWithMetadata
   [       OK ] TestLocalFile.OpenWithMetadata (1 ms)
   [----------] 1 test from TestLocalFile (1 ms total)
   
   [----------] 1 test from TestFileReaderAdHoc
   [ RUN      ] TestFileReaderAdHoc.NationDictTruncatedDataPage
   [       OK ] TestFileReaderAdHoc.NationDictTruncatedDataPage (1 ms)
   [----------] 1 test from TestFileReaderAdHoc (1 ms total)
   
   [----------] 1 test from TestDumpWithLocalFile
   [ RUN      ] TestDumpWithLocalFile.DumpOutput
   [       OK ] TestDumpWithLocalFile.DumpOutput (0 ms)
   [----------] 1 test from TestDumpWithLocalFile (0 ms total)
   
   [----------] 1 test from TestJSONWithLocalFile
   [ RUN      ] TestJSONWithLocalFile.JSONOutput
   [       OK ] TestJSONWithLocalFile.JSONOutput (0 ms)
   [----------] 1 test from TestJSONWithLocalFile (0 ms total)
   
   [----------] 3 tests from TestFileReader
   [ RUN      ] TestFileReader.BufferedReadsWithDictionary
   [       OK ] TestFileReader.BufferedReadsWithDictionary (2 ms)
   [ RUN      ] TestFileReader.BufferedReads
   [       OK ] TestFileReader.BufferedReads (14 ms)
   [ RUN      ] TestFileReader.TestOpenErrors
   [       OK ] TestFileReader.TestOpenErrors (1 ms)
   [----------] 3 tests from TestFileReader (18 ms total)
   
   [----------] 7 tests from TestStreamReader
   [ RUN      ] TestStreamReader.DefaultConstructed
   [       OK ] TestStreamReader.DefaultConstructed (28 ms)
   [ RUN      ] TestStreamReader.TypeChecking
   [       OK ] TestStreamReader.TypeChecking (28 ms)
   [ RUN      ] TestStreamReader.ValueChecking
   [       OK ] TestStreamReader.ValueChecking (34 ms)
   [ RUN      ] TestStreamReader.ReadRequiredFieldAsOptionalField
   [       OK ] TestStreamReader.ReadRequiredFieldAsOptionalField (35 ms)
   [ RUN      ] TestStreamReader.SkipRows
   [       OK ] TestStreamReader.SkipRows (36 ms)
   [ RUN      ] TestStreamReader.SkipAllRows
   [       OK ] TestStreamReader.SkipAllRows (27 ms)
   [ RUN      ] TestStreamReader.SkipColumns
   [       OK ] TestStreamReader.SkipColumns (45 ms)
   [----------] 7 tests from TestStreamReader (235 ms total)
   
   [----------] 2 tests from TestOptionalFields
   [ RUN      ] TestOptionalFields.ValueChecking
   [       OK ] TestOptionalFields.ValueChecking (41 ms)
   [ RUN      ] TestOptionalFields.ReadOptionalFieldAsRequiredField
   [       OK ] TestOptionalFields.ReadOptionalFieldAsRequiredField (55 ms)
   [----------] 2 tests from TestOptionalFields (97 ms total)
   
   [----------] 3 tests from TestReadingDataFiles
   [ RUN      ] TestReadingDataFiles.AllTypesPlain
   [       OK ] TestReadingDataFiles.AllTypesPlain (0 ms)
   [ RUN      ] TestReadingDataFiles.Int32Decimal
   [       OK ] TestReadingDataFiles.Int32Decimal (0 ms)
   [ RUN      ] TestReadingDataFiles.Int64Decimal
   [       OK ] TestReadingDataFiles.Int64Decimal (0 ms)
   [----------] 3 tests from TestReadingDataFiles (1 ms total)
   
   [----------] 6 tests from Lz4CodecTests/TestCodec
   [ RUN      ] Lz4CodecTests/TestCodec.SmallFileMetadataAndValues/LegacyLZ4Hadoop
   [       OK ] Lz4CodecTests/TestCodec.SmallFileMetadataAndValues/LegacyLZ4Hadoop (0 ms)
   [ RUN      ] Lz4CodecTests/TestCodec.SmallFileMetadataAndValues/LegacyLZ4NonHadoop
   [       OK ] Lz4CodecTests/TestCodec.SmallFileMetadataAndValues/LegacyLZ4NonHadoop (0 ms)
   [ RUN      ] Lz4CodecTests/TestCodec.SmallFileMetadataAndValues/LZ4Raw
   [       OK ] Lz4CodecTests/TestCodec.SmallFileMetadataAndValues/LZ4Raw (0 ms)
   [ RUN      ] Lz4CodecTests/TestCodec.LargeFileValues/LegacyLZ4Hadoop
   [       OK ] Lz4CodecTests/TestCodec.LargeFileValues/LegacyLZ4Hadoop (0 ms)
   [ RUN      ] Lz4CodecTests/TestCodec.LargeFileValues/LegacyLZ4NonHadoop
   /home/kou/work/cpp/arrow.kou/cpp/src/parquet/reader_test.cc:950: Skipped
   Larger data file not available for this codec
   [  SKIPPED ] Lz4CodecTests/TestCodec.LargeFileValues/LegacyLZ4NonHadoop (0 ms)
   [ RUN      ] Lz4CodecTests/TestCodec.LargeFileValues/LZ4Raw
   [       OK ] Lz4CodecTests/TestCodec.LargeFileValues/LZ4Raw (0 ms)
   [----------] 6 tests from Lz4CodecTests/TestCodec (2 ms total)
   
   [----------] Global test environment tear-down
   [==========] 82 tests from 27 test suites ran. (440 ms total)
   [  PASSED  ] 81 tests.
   [  SKIPPED ] 1 test, listed below:
   [  SKIPPED ] Lz4CodecTests/TestCodec.LargeFileValues/LegacyLZ4NonHadoop
   ```
   
   </details>
   
   Which test does use this method?



##########
cpp/src/parquet/reader_test.cc:
##########
@@ -127,6 +127,89 @@ void CheckRowGroupMetadata(const RowGroupMetaData* rg_metadata,
   }
 }
 
+class TestBooleanRLE : public ::testing::Test {
+ public:
+  void SetUp() {
+    reader_ = ParquetFileReader::OpenFile(data_file("rle_boolean_encoding.parquet"));
+  }
+
+  void TearDown() {}
+
+ protected:
+  std::unique_ptr<ParquetFileReader> reader_;
+};
+
+TEST_F(TestBooleanRLE, TestBooleanScanner) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto scanner = std::make_shared<BoolScanner>(group->Column(0));
+
+  bool val = false;
+  bool is_null = false;
+  for (int i = 0; i < 8; i++) {
+    ASSERT_TRUE(scanner->HasNext());
+    ASSERT_TRUE(scanner->NextValue(&val, &is_null));
+
+    // For this file, 3rd index value is null
+    if (i == 2) {
+      ASSERT_TRUE(is_null);
+    } else {
+      ASSERT_FALSE(is_null);
+    }
+  }
+
+  ASSERT_FALSE(scanner->HasNext());
+  ASSERT_FALSE(scanner->NextValue(&val, &is_null));
+}
+
+TEST_F(TestBooleanRLE, TestBatchRead) {
+  auto group = reader_->RowGroup(0);
+
+  // column 0, id
+  auto col = std::dynamic_pointer_cast<BoolReader>(group->Column(0));
+
+  // This file only has 8 rows
+  ASSERT_EQ(8, reader_->metadata()->num_rows());
+  // This file only has 1 row group
+  ASSERT_EQ(1, reader_->metadata()->num_row_groups());
+  // Size of the metadata is 106 bytes
+  ASSERT_EQ(106, reader_->metadata()->size());
+  // This row group must have 8 rows
+  ASSERT_EQ(8, group->metadata()->num_rows());
+
+  // Check if the column is encoded with RLE
+  auto col_chunk = group->metadata()->ColumnChunk(0);
+  ASSERT_TRUE(std::find(col_chunk->encodings().begin(), col_chunk->encodings().end(),
+                        Encoding::RLE) != col_chunk->encodings().end());
+
+  // Assert column has values to be read
+  ASSERT_TRUE(col->HasNext());
+  int64_t curr_batch_read = 0;
+
+  const int16_t batch_size = 8;
+  int16_t def_levels[batch_size];
+  int16_t rep_levels[batch_size];
+  bool values[batch_size - 1];
+
+  auto levels_read =
+      col->ReadBatch(batch_size, def_levels, rep_levels, values, &curr_batch_read);
+  ASSERT_EQ(batch_size, levels_read);
+
+  // Since one value is a null value, expect batches read to be one less than indicated
+  // batch_size
+  ASSERT_EQ(batch_size - 1, curr_batch_read);
+
+  // 3rd index is null value
+  ASSERT_THAT(def_levels, testing::ElementsAre(1, 1, 0, 1, 1, 1, 1, 1));
+
+  // Validate inserted data is as expected
+  ASSERT_THAT(values, testing::ElementsAre(1, 0, 1, 1, 0, 0, 1));

Review Comment:
   I used `gdb` and watched `RleDecoder::NextCounts()`. All calls use the `is_literal` clause:
   
   https://github.com/apache/arrow/blob/8daa7a4ed5629c0020dadf7325a6b523bdfc62e9/cpp/src/arrow/util/rle_encoding.h#L653-L656
   
   It seems that the test input have only `<bit-packed-run>` case.
   
   Are all boolean values (1/0) use `<bit-packed-run>` because they can be represented with 1 bit?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977099469


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   Hi @kou , 
   My bad again, I assumed `col->ReadBatch` , will internally call `decoder_->GetBatch`. 
   
   This method is covered in test `VectorBooleanTest`
   I went looking into the history why a boolean decoder has bits written to `uint8_t` datatype. This was introduced in this [commit](https://github.com/apache/arrow/commit/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc#diff-61373111f1c45b61c1232091a83f44e66895cf9d27b264d9cbfceac45ff87941). 
   
   This was introduced for testing purposes. The data is encoded with data from `vector<bool>` however it not possible to get reference for `bool` from `vector<bool>` which is needed to pass to Decode function
   `int Decode(bool* buffer, int max_values)
   `. [Link](https://en.wikipedia.org/w/index.php?title=Sequence_container_(C%2B%2B)&oldid=767869909#Specialization_for_bool) . 
   To bypass we introduced a reader with `uint_8` datatype. 
   
   I think a refactor on this is due, which is my next plan, that we replace `vector<bool>` with `bool[]` in test case and remove the class `BooleanDecoder`
   
   Proposed fix is here. Not including in this PR for compartmentalization. 
   
   ```diff
   diff --git a/cpp/src/parquet/encoding.cc b/cpp/src/parquet/encoding.cc
   index a862743d5..4a4646724 100644
   --- a/cpp/src/parquet/encoding.cc
   +++ b/cpp/src/parquet/encoding.cc
   @@ -1147,8 +1147,7 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
    }
    
    class PlainBooleanDecoder : public DecoderImpl,
   -                            virtual public TypedDecoder<BooleanType>,
   -                            virtual public BooleanDecoder {
   +                            virtual public TypedDecoder<BooleanType> {
     public:
      explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
      void SetData(int num_values, const uint8_t* data, int len) override;
   @@ -2358,7 +2357,7 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
    // ----------------------------------------------------------------------
    // RLE_BOOLEAN_DECODER
    
   -class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
   +class RleBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
     public:
      explicit RleBooleanDecoder(const ColumnDescriptor* descr)
          : DecoderImpl(descr, Encoding::RLE) {}
   diff --git a/cpp/src/parquet/encoding.h b/cpp/src/parquet/encoding.h
   index b9ca7a7ee..c32da9979 100644
   --- a/cpp/src/parquet/encoding.h
   +++ b/cpp/src/parquet/encoding.h
   @@ -65,7 +65,7 @@ using FLBAEncoder = TypedEncoder<FLBAType>;
    template <typename DType>
    class TypedDecoder;
    
   -class BooleanDecoder;
   +using BooleanDecoder = TypedDecoder<BooleanType>;
    using Int32Decoder = TypedDecoder<Int32Type>;
    using Int64Decoder = TypedDecoder<Int64Type>;
    using Int96Decoder = TypedDecoder<Int96Type>;
   @@ -394,12 +394,6 @@ class DictDecoder : virtual public TypedDecoder<DType> {
    // ----------------------------------------------------------------------
    // TypedEncoder specializations, traits, and factory functions
    
   -class BooleanDecoder : virtual public TypedDecoder<BooleanType> {
   - public:
   -  using TypedDecoder<BooleanType>::Decode;
   -  virtual int Decode(uint8_t* buffer, int max_values) = 0;
   -};
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r977099469


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   My bad again, I assumed `col->ReadBatch` , will internally call `decoder_->GetBatch`. 
   
   This method is covered in test `VectorBooleanTest`
   I went looking into the history why a boolean decoder has bits written to `uint8_t` datatype. This was introduced in this [commit](https://github.com/apache/arrow/commit/3d435e4f8d5fb7a54a4a9d285e1a42d60186d8dc#diff-61373111f1c45b61c1232091a83f44e66895cf9d27b264d9cbfceac45ff87941). 
   
   This was introduced for testing purposes. The data is encoded with data from `vector<bool>` however it not possible to get reference for `bool` from `vector<bool>` which is needed to pass to Decode function
   `int Decode(bool* buffer, int max_values)
   `. [Link](https://en.wikipedia.org/w/index.php?title=Sequence_container_(C%2B%2B)&oldid=767869909#Specialization_for_bool) . 
   To bypass we introduced a reader with `uint_8` datatype. 
   
   I think a refactor on this is due, which is my next plan, that we replace `vector<bool>` with `bool[]` in test case and remove the class `BooleanDecoder`
   
   Proposed fix is here. Not including in this PR for compartmentalization. 
   
   ```c++
   diff --git a/cpp/src/parquet/encoding.cc b/cpp/src/parquet/encoding.cc
   index a862743d5..4a4646724 100644
   --- a/cpp/src/parquet/encoding.cc
   +++ b/cpp/src/parquet/encoding.cc
   @@ -1147,8 +1147,7 @@ int PlainDecoder<DType>::Decode(T* buffer, int max_values) {
    }
    
    class PlainBooleanDecoder : public DecoderImpl,
   -                            virtual public TypedDecoder<BooleanType>,
   -                            virtual public BooleanDecoder {
   +                            virtual public TypedDecoder<BooleanType> {
     public:
      explicit PlainBooleanDecoder(const ColumnDescriptor* descr);
      void SetData(int num_values, const uint8_t* data, int len) override;
   @@ -2358,7 +2357,7 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
    // ----------------------------------------------------------------------
    // RLE_BOOLEAN_DECODER
    
   -class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
   +class RleBooleanDecoder : public DecoderImpl, virtual public TypedDecoder<BooleanType> {
     public:
      explicit RleBooleanDecoder(const ColumnDescriptor* descr)
          : DecoderImpl(descr, Encoding::RLE) {}
   diff --git a/cpp/src/parquet/encoding.h b/cpp/src/parquet/encoding.h
   index b9ca7a7ee..c32da9979 100644
   --- a/cpp/src/parquet/encoding.h
   +++ b/cpp/src/parquet/encoding.h
   @@ -65,7 +65,7 @@ using FLBAEncoder = TypedEncoder<FLBAType>;
    template <typename DType>
    class TypedDecoder;
    
   -class BooleanDecoder;
   +using BooleanDecoder = TypedDecoder<BooleanType>;
    using Int32Decoder = TypedDecoder<Int32Type>;
    using Int64Decoder = TypedDecoder<Int64Type>;
    using Int96Decoder = TypedDecoder<Int96Type>;
   @@ -394,12 +394,6 @@ class DictDecoder : virtual public TypedDecoder<DType> {
    // ----------------------------------------------------------------------
    // TypedEncoder specializations, traits, and factory functions
    
   -class BooleanDecoder : virtual public TypedDecoder<BooleanType> {
   - public:
   -  using TypedDecoder<BooleanType>::Decode;
   -  virtual int Decode(uint8_t* buffer, int max_values) = 0;
   -};
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r973666860


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,83 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl,  virtual public BooleanDecoder {
+
+  public:
+    explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+            :DecoderImpl(descr, Encoding::RLE) {}
+
+    void SetData(int num_values, const uint8_t* data, int len) override {
+      num_values_ = num_values;
+      int32_t num_bytes = 0;
+
+      if (len < 4) {
+        throw ParquetException("Received invalid length (corrupt data page?)");
+      }
+      // Load the first 4 bytes, which indicates the legnth
+      num_bytes = ::arrow::util::SafeLoadAs<int32_t>(data);
+      if (num_bytes < 0 || num_bytes > len - 4) {
+        throw ParquetException("Received invalid number of bytes (corrupt data page?)");

Review Comment:
   Modified error message to show invalid bytes value. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] kou commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
kou commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1263261877

   Could you confirm this failure? https://github.com/apache/arrow/actions/runs/3155990217/jobs/5136453754#step:6:3035
   
   ```text
   [----------] 2 tests from TestBooleanRLE
   [ RUN      ] TestBooleanRLE.TestBooleanScanner
   [       OK ] TestBooleanRLE.TestBooleanScanner (4 ms)
   [ RUN      ] TestBooleanRLE.TestBatchRead
   =================================================================
   ==24088==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd9a55539f at pc 0x7f4c6b9f033d bp 0x7ffd9a5528b0 sp 0x7ffd9a5528a8
   WRITE of size 1 at 0x7ffd9a55539f thread T0
       #0 0x7f4c6b9f033c in void arrow::bit_util::detail::GetValue_<bool>(int, bool*, int, unsigned char const*, int*, int*, unsigned long*) /arrow/cpp/src/arrow/util/bit_stream_utils.h:279:6
       #1 0x7f4c6b9edb05 in int arrow::bit_util::BitReader::GetBatch<bool>(int, bool*, int) /arrow/cpp/src/arrow/util/bit_stream_utils.h:338:7
       #2 0x7f4c6b9ec0e6 in int arrow::util::RleDecoder::GetBatch<bool>(bool*, int) /arrow/cpp/src/arrow/util/rle_encoding.h:320:37
       #3 0x7f4c6b9dd98d in parquet::(anonymous namespace)::RleBooleanDecoder::Decode(bool*, int) /arrow/cpp/src/parquet/encoding.cc:2368:19
       #4 0x7f4c6b676a35 in parquet::(anonymous namespace)::ColumnReaderImplBase<parquet::PhysicalType<(parquet::Type::type)0> >::ReadValues(long, bool*) /arrow/cpp/src/parquet/column_reader.cc:576:45
       #5 0x7f4c6b65c99c in parquet::(anonymous namespace)::TypedColumnReaderImpl<parquet::PhysicalType<(parquet::Type::type)0> >::ReadBatch(long, short*, short*, bool*, long*) /arrow/cpp/src/parquet/column_reader.cc:1044:24
       #6 0x7a3c21 in parquet::TestBooleanRLE_TestBatchRead_Test::TestBody() /arrow/cpp/src/parquet/reader_test.cc:240:14
       #7 0x7f4c6d436a3a in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:2607:10
       #8 0x7f4c6d41b569 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:2643:14
       #9 0x7f4c6d3f4c42 in testing::Test::Run() /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:2682:5
       #10 0x7f4c6d3f5a08 in testing::TestInfo::Run() /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:2861:11
       #11 0x7f4c6d3f6223 in testing::TestSuite::Run() /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:3015:28
       #12 0x7f4c6d407004 in testing::internal::UnitTestImpl::RunAllTests() /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:5855:44
       #13 0x7f4c6d4398da in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:2607:10
       #14 0x7f4c6d41dd89 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:2643:14
       #15 0x7f4c6d406b60 in testing::UnitTest::Run() /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest.cc:5438:10
       #16 0x7f4c6d471210 in RUN_ALL_TESTS() /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/include/gtest/gtest.h:2490:46
       #17 0x7f4c6d4711ec in main /build/cpp/googletest_ep-prefix/src/googletest_ep/googletest/src/gtest_main.cc:52:10
       #18 0x7f4c4efc5082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082)
       #19 0x47330d in _start (/build/cpp/debug/parquet-reader-test+0x47330d)
   
   Address 0x7ffd9a55539f is located in stack of thread T0 at offset 1503 in frame
       #0 0x79e64f in parquet::TestBooleanRLE_TestBatchRead_Test::TestBody() /arrow/cpp/src/parquet/reader_test.cc:188
   
     This frame has 104 object(s):
       [32, 36) 'nvalues' (line 189)
       [48, 52) 'num_row_groups' (line 190)
       [64, 68) 'metadata_size' (line 191)
       [80, 96) 'group' (line 193)
       [112, 128) 'col' (line 196)
       [144, 160) 'ref.tmp' (line 196)
       [176, 192) 'gtest_ar' (line 199)
       [208, 216) 'ref.tmp9' (line 199)
       [240, 256) 'ref.tmp10' (line 199)
       [272, 280) 'ref.tmp30' (line 199)
       [304, 312) 'ref.tmp33' (line 199)
       [336, 352) 'gtest_ar47' (line 201)
       [368, 372) 'ref.tmp48' (line 201)
       [384, 400) 'ref.tmp49' (line 201)
       [416, 424) 'ref.tmp73' (line 201)
       [448, 456) 'ref.tmp76' (line 201)
       [480, 496) 'gtest_ar94' (line 203)
       [512, 516) 'ref.tmp95' (line 203)
       [528, 544) 'ref.tmp96' (line 203)
       [560, 568) 'ref.tmp120' (line 203)
       [592, 600) 'ref.tmp123' (line 203)
       [624, 640) 'gtest_ar141' (line 205)
       [656, 664) 'ref.tmp142' (line 205)
       [688, 696) 'ref.tmp160' (line 205)
       [720, 728) 'ref.tmp163' (line 205)
       [752, 760) 'col_chunk' (line 208)
       [784, 800) 'gtest_ar_' (line 209)
       [816, 817) 'ref.tmp190' (line 209)
       [832, 840) 'ref.tmp191' (line 209)
       [864, 872) 'agg.tmp'
       [896, 904) 'agg.tmp201'
       [928, 932) 'ref.tmp211' (line 209)
       [944, 952) 'ref.tmp218' (line 209)
       [976, 984) 'ref.tmp240' (line 209)
       [1008, 1016) 'ref.tmp243' (line 209)
       [1040, 1072) 'ref.tmp244' (line 209)
       [1104, 1120) 'gtest_ar_265' (line 213)
       [1136, 1137) 'ref.tmp266' (line 213)
       [1152, 1160) 'ref.tmp281' (line 213)
       [1184, 1192) 'ref.tmp284' (line 213)
       [1216, 1248) 'ref.tmp285' (line 213)
       [1280, 1288) 'curr_batch_read' (line 214)
       [1312, 1314) 'batch_size' (line 216)
       [1328, 1362) 'def_levels' (line 218)
       [1408, 1442) 'rep_levels' (line 219)
       [1488, 1503) 'values' (line 220) <== Memory access at offset 1503 overflows this variable
       [1520, 1528) 'levels_read' (line 222)
       [1552, 1568) 'gtest_ar316' (line 224)
       [1584, 1592) 'ref.tmp324' (line 224)
       [1616, 1624) 'ref.tmp327' (line 224)
       [1648, 1664) 'gtest_ar345' (line 228)
       [1680, 1684) 'ref.tmp346' (line 228)
       [1696, 1704) 'ref.tmp355' (line 228)
       [1728, 1736) 'ref.tmp358' (line 228)
       [1760, 1776) 'gtest_ar376' (line 231)
       [1792, 1860) 'ref.tmp377' (line 231)
       [1904, 1972) 'agg.tmp378'
       [2016, 2020) 'ref.tmp379' (line 231)
       [2032, 2036) 'ref.tmp380' (line 231)
       [2048, 2052) 'ref.tmp381' (line 231)
       [2064, 2068) 'ref.tmp382' (line 231)
       [2080, 2084) 'ref.tmp383' (line 231)
       [2096, 2100) 'ref.tmp384' (line 231)
       [2112, 2116) 'ref.tmp385' (line 231)
       [2128, 2132) 'ref.tmp386' (line 231)
       [2144, 2148) 'ref.tmp387' (line 231)
       [2160, 2164) 'ref.tmp388' (line 231)
       [2176, 2180) 'ref.tmp389' (line 231)
       [2192, 2196) 'ref.tmp390' (line 231)
       [2208, 2212) 'ref.tmp391' (line 231)
       [2224, 2228) 'ref.tmp392' (line 231)
       [2240, 2244) 'ref.tmp393' (line 231)
       [2256, 2260) 'ref.tmp394' (line 231)
       [2272, 2276) 'ref.tmp395' (line 231)
       [2288, 2296) 'ref.tmp423' (line 231)
       [2320, 2328) 'ref.tmp426' (line 231)
       [2352, 2368) 'gtest_ar444' (line 235)
       [2384, 2444) 'ref.tmp445' (line 235)
       [2480, 2540) 'agg.tmp446'
       [2576, 2580) 'ref.tmp447' (line 235)
       [2592, 2596) 'ref.tmp448' (line 235)
       [2608, 2612) 'ref.tmp449' (line 235)
       [2624, 2628) 'ref.tmp450' (line 235)
       [2640, 2644) 'ref.tmp451' (line 235)
       [2656, 2660) 'ref.tmp452' (line 235)
       [2672, 2676) 'ref.tmp453' (line 235)
       [2688, 2692) 'ref.tmp454' (line 235)
       [2704, 2708) 'ref.tmp455' (line 235)
       [2720, 2724) 'ref.tmp456' (line 235)
       [2736, 2740) 'ref.tmp457' (line 235)
       [2752, 2756) 'ref.tmp458' (line 235)
       [2768, 2772) 'ref.tmp459' (line 235)
       [2784, 2788) 'ref.tmp460' (line 235)
       [2800, 2804) 'ref.tmp461' (line 235)
       [2816, 2824) 'ref.tmp487' (line 235)
       [2848, 2856) 'ref.tmp490' (line 235)
       [2880, 2896) 'gtest_ar519' (line 241)
       [2912, 2920) 'ref.tmp527' (line 241)
       [2944, 2952) 'ref.tmp530' (line 241)
       [2976, 2992) 'gtest_ar_552' (line 245)
       [3008, 3009) 'ref.tmp553' (line 245)
       [3024, 3032) 'ref.tmp570' (line 245)
       [3056, 3064) 'ref.tmp573' (line 245)
       [3088, 3120) 'ref.tmp574' (line 245)
   HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
         (longjmp and C++ exceptions *are* supported)
   SUMMARY: AddressSanitizer: stack-buffer-overflow /arrow/cpp/src/arrow/util/bit_stream_utils.h:279:6 in void arrow::bit_util::detail::GetValue_<bool>(int, bool*, int, unsigned char const*, int*, int*, unsigned long*)
   Shadow bytes around the buggy address:
     0x1000334a2a20: f8 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 f8 f2 f8 f2
     0x1000334a2a30: f2 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f8 f8 f8 f2 f2
     0x1000334a2a40: f2 f2 f8 f8 f2 f2 f8 f2 f8 f2 f2 f2 f8 f2 f2 f2
     0x1000334a2a50: f8 f8 f8 f8 f2 f2 f2 f2 00 f2 f2 f2 02 f2 00 00
     0x1000334a2a60: 00 00 02 f2 f2 f2 f2 f2 00 00 00 00 02 f2 f2 f2
   =>0x1000334a2a70: f2 f2 00[07]f2 f2 00 f2 f2 f2 f8 f8 f2 f2 f8 f2
     0x1000334a2a80: f2 f2 f8 f2 f2 f2 f8 f8 f2 f2 f8 f2 f8 f2 f2 f2
     0x1000334a2a90: f8 f2 f2 f2 f8 f8 f2 f2 f8 f8 f8 f8 f8 f8 f8 f8
     0x1000334a2aa0: f8 f2 f2 f2 f2 f2 00 00 00 00 00 00 00 00 04 f2
     0x1000334a2ab0: f2 f2 f2 f2 f8 f2 f8 f2 f8 f2 f8 f2 f8 f2 f8 f2
     0x1000334a2ac0: f8 f2 f8 f2 f8 f2 f8 f2 f8 f2 f8 f2 f8 f2 f8 f2
   Shadow byte legend (one shadow byte represents 8 application bytes):
     Addressable:           00
     Partially addressable: 01 02 03 04 05 06 07
     Heap left redzone:       fa
     Freed heap region:       fd
     Stack left redzone:      f1
     Stack mid redzone:       f2
     Stack right redzone:     f3
     Stack after return:      f5
     Stack use after scope:   f8
     Global redzone:          f9
     Global init order:       f6
     Poisoned by user:        f7
     Container overflow:      fc
     Array cookie:            ac
     Intra object redzone:    bb
     ASan internal:           fe
     Left alloca redzone:     ca
     Right alloca redzone:    cb
     Shadow gap:              cc
   ==24088==ABORTING
   /build/cpp/src/parquet
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on a diff in pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on code in PR #14147:
URL: https://github.com/apache/arrow/pull/14147#discussion_r984152096


##########
cpp/src/parquet/encoding.cc:
##########
@@ -2355,6 +2355,81 @@ class DeltaLengthByteArrayDecoder : public DecoderImpl,
   std::shared_ptr<ResizableBuffer> buffered_data_;
 };
 
+// ----------------------------------------------------------------------
+// RLE_BOOLEAN_DECODER
+
+class RleBooleanDecoder : public DecoderImpl, virtual public BooleanDecoder {
+ public:
+  explicit RleBooleanDecoder(const ColumnDescriptor* descr)
+      : DecoderImpl(descr, Encoding::RLE) {}
+
+  void SetData(int num_values, const uint8_t* data, int len) override {
+    num_values_ = num_values;
+    uint32_t num_bytes = 0;
+
+    if (len < 4) {
+      throw ParquetException("Received invalid length : " + std::to_string(len) +
+                             " (corrupt data page?)");
+    }
+    // Load the first 4 bytes in little-endian, which indicates the length
+    num_bytes =
+        ::arrow::bit_util::ToLittleEndian(::arrow::util::SafeLoadAs<uint32_t>(data));
+    if (num_bytes < 0 || num_bytes > static_cast<uint32_t>(len - 4)) {
+      throw ParquetException("Received invalid number of bytes : " +
+                             std::to_string(num_bytes) + " (corrupt data page?)");
+    }
+
+    auto decoder_data = data + 4;
+    decoder_ = std::make_shared<::arrow::util::RleDecoder>(decoder_data, num_bytes,
+                                                           /*bit_width=*/1);
+  }
+
+  int Decode(bool* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    int val = 0;
+
+    for (int i = 0; i < max_values; ++i) {
+      if (!decoder_->Get(&val)) {
+        throw ParquetException("Unable to parse bits for position (0 based) : " +
+                               std::to_string(i) + " (corrupt data page?)");
+      }
+      if (val) {
+        buffer[i] = true;
+      } else {
+        buffer[i] = false;
+      }
+    }
+    num_values_ -= max_values;
+    return max_values;
+  }
+
+  int Decode(uint8_t* buffer, int max_values) override {
+    max_values = std::min(max_values, num_values_);
+    if (decoder_->GetBatch(buffer, max_values) != max_values) {

Review Comment:
   Figured easier and better to work on this PR. Made above changes  in this PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE decode for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1263043836

   > Could you rebase and resolve the current conflict?
   
   Done. Rebased and resolved merge conflicts. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] sfc-gh-nthimmegowda commented on pull request #14147: ARROW-17450 : [C++][Parquet] Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
sfc-gh-nthimmegowda commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1251925862

   Hey @kou , 
   Really appreciate you taking the time to review this. Let me know if there are any more issues, and I will address all of them. 
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #14147: ARROW-17450 : Support RLE encoding for boolean datatype

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14147:
URL: https://github.com/apache/arrow/pull/14147#issuecomment-1248637449

   :warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org