[PR] [common] Reject out-of-range integral casts when extracting a variant scalar [paimon]

["zhuxiangyi (via GitHub)" <gi...@apache.org>]

zhuxiangyi opened a new pull request, #9809:
URL: https://github.com/apache/paimon/pull/9809

   ### Purpose
   
   Extracting a variant number into a narrower integral type goes through the generic `CastExecutors` rules (`Number::intValue`, `(int) DecimalUtils.castToIntegral(...)`), which wrap or saturate a value that does not fit. With `spark.sql.variant.pushVariantIntoScan=true` on Spark 4.1 the cast runs inside the Paimon reader, so the same query returns different numbers depending on the session flag:
   
   ```sql
   CREATE TABLE T (id INT, v VARIANT);
   INSERT INTO T VALUES (1, parse_json('{"n":99999999999,"d":1e30}')), (2, parse_json('{"n":7,"d":1.5}'));
   
   SELECT id, try_variant_get(v, '$.n', 'int'), try_variant_get(v, '$.d', 'bigint') FROM T ORDER BY id;
   -- pushVariantIntoScan=true : (1, 1215752191, 9223372036854775807), (2, 7, 1)
   -- pushVariantIntoScan=false: (1, NULL, NULL), (2, 7, 1)      <- Spark's TRY cast
   ```
   
   The shredded `typed_value` path (`BaseVariantReader.ScalarReader`) shares the problem through the same cast rules, e.g. a `BIGINT` typed value read as `INT`.
   
   This PR checks the truncated value against the target range before delegating to the cast rules, in a helper shared by `VariantGet` and the scalar reader, so an out-of-range number becomes an invalid cast (`NULL` for `try_variant_get`, an error for `variant_get`), matching Spark. A cast rule returning `null` (e.g. a `BIGINT` that overflows a `DECIMAL(p, s)` target) is now also reported as an invalid cast instead of being returned as a silent `NULL` under `failOnError=true`.
   
   In-range behaviour is unchanged: fractional parts are still truncated (`1.5` as `int` → `1`) and widening casts still succeed.
   
   ### Tests
   
   - `GenericVariantTest#testVariantGetIntegralOverflow`: LONG / DOUBLE / DECIMAL inputs into TINYINT..BIGINT, try mode returns `NULL`, strict mode throws; boundary values still cast.
   - `VariantShreddingReadTest#testReadIntegralOverflowAsInvalidCast`: parameterised over a plain file and a file shredded with `n BIGINT, d DOUBLE`, covering both `VariantGet` and the shredded scalar reader.
   - `VariantTestBase`: `try_variant_get` out-of-range extraction returns `NULL`; runs under all four Spark 4.x configurations (with/without pushdown, with/without inferred shredding). Verified on Spark 4.1.2: 116 tests pass.
   
   ### API and Format
   
   No.
   
   ### Documentation
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [common] Reject out-of-range integral casts when extracting a variant scalar [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #9809:
URL: https://github.com/apache/paimon/pull/9809


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [common] Reject out-of-range integral casts when extracting a variant scalar [paimon]

["zhuxiangyi (via GitHub)" <gi...@apache.org>]

zhuxiangyi commented on PR #9809:
URL: https://github.com/apache/paimon/pull/9809#issuecomment-5674468568

   Thanks for the review. Rebased onto master after #9818 landed (it touched `VariantGet` / `BaseVariantReader` and added tests next to the ones here); conflicts are resolved and the change is still a single commit. On the Spark 3 / Scala 2.13 failure: it was `RowTrackingTestBase` "Data Evolution: concurrent merge and small files compact" hitting an expired snapshot during a concurrent MERGE + compact, which this PR does not touch (Spark 3 does not run the Variant suites at all). CI is re-running on the rebased head and that job has not failed again so far. Locally after the rebase: `GenericVariantTest` / `VariantShreddingReadTest` pass and the four Spark 4.1 Variant suites pass (124 tests).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org