You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "CynicDog (via GitHub)" <gi...@apache.org> on 2026/05/16 12:26:08 UTC

[PR] Validate FIXED_LEN_BYTE_ARRAY length for DECIMAL and INTERVAL types [arrow-rs]

CynicDog opened a new pull request, #9985:
URL: https://github.com/apache/arrow-rs/pull/9985

   # Which issue does this PR close?
   
   - Closes #9984.
   
   # Rationale for this change
   
   `from_fixed_len_byte_array` in `parquet/src/arrow/schema/primitive.rs` does not validate `type_length`. `PrimitiveTypeBuilder::build()` enforces these constraints at construction (`schema/types.rs:477` for INTERVAL, `:565-580` for DECIMAL), but schemas decoded directly from Thrift bypass that path. As a result:
   
   - `DECIMAL` with `type_length` outside `1..=32` silently routed through `Decimal128` / `Decimal256` with bogus inputs.
   - `INTERVAL` with `type_length != 12` silently returned `Interval(DayTime)` regardless.
   
   The same function already rejects `FLOAT16` when `type_length != 2`. This PR mirrors that pattern for DECIMAL and INTERVAL, closing the TODO introduced in #1682.
   
   # What changes are included in this PR?
   
   - `check_decimal_length` helper rejects `type_length` outside `1..=32` for both `LogicalType::Decimal` and `ConvertedType::DECIMAL`.
   - Inline `type_length == 12` check for `ConvertedType::INTERVAL`.
   
   # Are these changes tested?
   
   Yes. Five new tests in `parquet/src/arrow/schema/primitive.rs::tests` cover invalid lengths (`{-1, 0, 33}` for DECIMAL, `{0, 11, 13}` for INTERVAL) and valid lengths (16 → `Decimal128`, 32 → `Decimal256`, 12 → `Interval(DayTime)`). To exercise the reader-side check, tests construct a valid `Type::PrimitiveType` via the builder and rewrite `type_length` on the resulting enum, simulating a schema decoded from Thrift.
   
   # Are there any user-facing changes?
   
   No public API changes. The behavior change is reader-side: schemas with out-of-range `type_length` for DECIMAL/INTERVAL now return `ParquetError::General` instead of silently producing a wrong Arrow type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Validate FIXED_LEN_BYTE_ARRAY length for DECIMAL and INTERVAL types [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9985:
URL: https://github.com/apache/arrow-rs/pull/9985#discussion_r3283446161


##########
parquet/src/arrow/schema/primitive.rs:
##########
@@ -168,6 +168,16 @@ fn decimal_256_type(scale: i32, precision: i32) -> Result<DataType> {
     Ok(DataType::Decimal256(precision, scale))
 }
 
+fn check_decimal_length(type_length: i32) -> Result<()> {
+    // Arrow's widest decimal is Decimal256, which needs at most 32 bytes.
+    if !(1..=32).contains(&type_length) {

Review Comment:
   I found this a confusing way to check bounds -- I would expect something like
   ```rust
   if type_length < 1 || type_length > 32 {
   ```



##########
parquet/src/arrow/schema/primitive.rs:
##########
@@ -313,23 +323,29 @@ fn from_fixed_len_byte_array(
     precision: i32,
     type_length: i32,
 ) -> Result<DataType> {
-    // TODO: This should check the type length for the decimal and interval types
     match (info.logical_type_ref(), info.converted_type()) {
         (Some(LogicalType::Decimal { scale, precision }), _) => {
+            check_decimal_length(type_length)?;

Review Comment:
   should this be checking exact lengths? The only valid lengths in this case are 16 and 32 , right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Validate FIXED_LEN_BYTE_ARRAY length for DECIMAL and INTERVAL types [arrow-rs]

Posted by "CynicDog (via GitHub)" <gi...@apache.org>.
CynicDog commented on code in PR #9985:
URL: https://github.com/apache/arrow-rs/pull/9985#discussion_r3288194748


##########
parquet/src/arrow/schema/primitive.rs:
##########
@@ -168,6 +168,16 @@ fn decimal_256_type(scale: i32, precision: i32) -> Result<DataType> {
     Ok(DataType::Decimal256(precision, scale))
 }
 
+fn check_decimal_length(type_length: i32) -> Result<()> {
+    // Arrow's widest decimal is Decimal256, which needs at most 32 bytes.
+    if !(1..=32).contains(&type_length) {

Review Comment:
   Makes sense, updated!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Validate FIXED_LEN_BYTE_ARRAY length for DECIMAL and INTERVAL types [arrow-rs]

Posted by "CynicDog (via GitHub)" <gi...@apache.org>.
CynicDog commented on code in PR #9985:
URL: https://github.com/apache/arrow-rs/pull/9985#discussion_r3288159774


##########
parquet/src/arrow/schema/primitive.rs:
##########
@@ -313,23 +323,29 @@ fn from_fixed_len_byte_array(
     precision: i32,
     type_length: i32,
 ) -> Result<DataType> {
-    // TODO: This should check the type length for the decimal and interval types
     match (info.logical_type_ref(), info.converted_type()) {
         (Some(LogicalType::Decimal { scale, precision }), _) => {
+            check_decimal_length(type_length)?;

Review Comment:
   Good catch on the wording! You're totally right—any byte length is valid here, not just 16 or 32. 
   
   As per the [Parquet spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal) for `fixed_len_byte_array`, the precision is simply limited by the array size. As long as the precision fits within the byte length, it's completely valid.
   
   That upper bound of 32 we were looking at is actually an Arrow constraint (since `Decimal256` is 256 bits, which caps it at 32 bytes), rather than a limitation from Parquet itself. Checking for a range of `1..=32` makes perfect sense here, especially since we definitely see perfectly legitimate Parquet files out in the wild with byte lengths like 4 or 7.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Validate FIXED_LEN_BYTE_ARRAY length for DECIMAL and INTERVAL types [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #9985:
URL: https://github.com/apache/arrow-rs/pull/9985


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org