You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Joob1n (via GitHub)" <gi...@apache.org> on 2026/09/15 07:33:29 UTC

[PR] fix(parquet): decode DECIMAL page statistics by physical type [paimon-cpp]

Joob1n opened a new pull request, #346:
URL: https://github.com/apache/paimon-cpp/pull/346

   ### Purpose
   
   Linked issue: close #345
   
   `ColumnIndexFilter::CompareEncodedWithLiteral` inferred the physical type of DECIMAL page statistics from the length of the encoded min/max — 4 bytes meant INT32, 8 bytes meant INT64 (both decoded with a native-endian `memcpy`), anything else was decoded as big-endian two's complement.
   
   Parquet allows DECIMAL as FIXED_LEN_BYTE_ARRAY for any precision, with length `ceil((precision * log2(10) + 1) / 8)`. Precision 7-9 yields exactly 4 bytes and precision 17-18 exactly 8, so those columns took the INT32/INT64 branch and had their big-endian bounds read little-endian. Page min `1.00` (unscaled 100, stored as `00 00 00 64`) read back as 1,677,721,600, so `col < 5.00` answered "no" and dropped a page that does match. Page-index filtering is pruning, not filtering, so nothing downstream brings those rows back.
   
   This passes the `::parquet::SchemaDescriptor` down from `FileReaderWrapper::CalculateFilteredRowRanges` through `ColumnIndexFilter` and dispatches on the column's physical type, the way `parquet_stats_extractor.cpp` already does for row-group statistics. INT32/INT64 keep the plain little-endian decode; FIXED_LEN_BYTE_ARRAY and BYTE_ARRAY go through `Decimal::FromUnscaledBytes`. When no schema is available the comparison is skipped, so no page is pruned.
   
   Paimon's own writers use INT32 for precision <= 9 and INT64 for precision <= 18, so this only reaches users through files written elsewhere: Arrow with the default `store_decimal_as_integer = false`, Spark with `spark.sql.parquet.writeLegacyFormat = true`, or Hive.
   
   ### Tests
   
   New cases in `src/paimon/format/parquet/column_index_filter_test.cpp`:
   
   - `ColumnIndexFilterTest.DecimalFixedLenByteArrayPages` — writes DECIMAL(9,2) and DECIMAL(18,2) columns as FIXED_LEN_BYTE_ARRAY with the page index enabled, asserts the writer really produced FLBA of length 4 and 8, then checks that `<`, `>` and `=` keep exactly the pages holding matching rows and prune the rest. Both cases fail on the previous decoding logic and pass with this change.
   - `ColumnIndexFilterTest.DecimalWithoutSchemaKeepsAllPages` — with no schema descriptor, no page is pruned.
   
   Run locally (macOS arm64, Debug, `-Dfmt_SOURCE=BUNDLED`):
   
   - `./build/debug/paimon-parquet-format-test --gtest_filter='ColumnIndexFilterTest.*:RowRangesTest.*'` — 36/36 pass.
   - Full `paimon-parquet-format-test` (225 cases, run one per process): 147 pass, 78 fail. Those 78 fail identically on unmodified `main` — they abort in fixture setup at `arrow/ipc/json_simple.cc:156 DCHECK(*out)` inside `ArrayFromJSON`, which is unrelated to this change and Debug-only.
   
   ### API and Format
   
   No. `ColumnIndexFilter::CalculateRowRanges` gains a `const ::parquet::SchemaDescriptor*` parameter, but the class is internal to `src/paimon/format/parquet/` and is not part of the public headers in `include/`. No storage format or protocol change.
   
   ### Documentation
   
   No.
   
   ### Generative AI tooling
   
   Generated-by: Claude Code (Claude Opus 5)
   
   https://claude.ai/code/session_01NnX2Q5dAwrmeWYmf9n1mCC
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(parquet): decode DECIMAL page statistics by physical type [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 commented on code in PR #346:
URL: https://github.com/apache/paimon-cpp/pull/346#discussion_r4013210829


##########
src/paimon/format/parquet/column_index_filter.cpp:
##########
@@ -68,7 +70,8 @@ Result<RowRanges> ColumnIndexFilter::VisitPredicate(
 Result<RowRanges> ColumnIndexFilter::VisitLeafPredicate(
     const std::shared_ptr<LeafPredicate>& leaf_predicate,
     const std::map<std::string, int32_t>& column_name_to_index, int64_t row_group_row_count,
-    ::parquet::RowGroupPageIndexReader* rg_page_index_reader) {
+    ::parquet::RowGroupPageIndexReader* rg_page_index_reader,
+    const ::parquet::SchemaDescriptor* schema) {
     const std::string& field_name = leaf_predicate->FieldName();

Review Comment:
   Could you please keep output parameters at the end? If `schema` is a `const*`, please place it in the middle.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(parquet): decode DECIMAL page statistics by physical type [paimon-cpp]

Posted by "Joob1n (via GitHub)" <gi...@apache.org>.
Joob1n commented on PR #346:
URL: https://github.com/apache/paimon-cpp/pull/346#issuecomment-5676531403

   @lxy-9602 @SteNicholas could you take a look when you have a moment?
   
   This is a small correctness fix in the page-index path: `ColumnIndexFilter` was guessing the physical type of DECIMAL page bounds from their encoded length, so a DECIMAL stored as a 4-byte (precision 7-9) or 8-byte (precision 17-18) FIXED_LEN_BYTE_ARRAY had its big-endian bounds read little-endian and matching pages were pruned away. The fix threads the `SchemaDescriptor` down to the leaf and dispatches on `physical_type()`, mirroring what `parquet_stats_extractor.cpp` already does for row-group statistics.
   
   The one API-shaped detail worth your eye: `ColumnIndexFilter::CalculateRowRanges` gains a `const ::parquet::SchemaDescriptor*` parameter (internal to `src/paimon/format/parquet/`, nothing in `include/`), and a null schema means the DECIMAL comparison is skipped so no page is pruned. Happy to change that to a hard requirement instead if you prefer.
   
   Details and test results are in the PR description; it closes #345.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix(parquet): decode DECIMAL page statistics by physical type [paimon-cpp]

Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 merged PR #346:
URL: https://github.com/apache/paimon-cpp/pull/346


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org