You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org> on 2026/08/07 15:30:32 UTC

[PR] [Fix][Transform-V2] Fix silent value corruption in Zeta SQL numeric functions [seatunnel]

SEPURI-SAI-KRISHNA opened a new pull request, #11697:
URL: https://github.com/apache/seatunnel/pull/11697

   ### Purpose of this pull request
   
   Closes #11696.
   
   `CEIL`, `CEILING`, `FLOOR`, `ROUND`, `TRUNC`, `TRUNCATE` and `MOD` in the Zeta SQL transform lost precision or overflowed silently, writing wrong values to the sink without raising any error. The job reports success, so the corruption is invisible until someone audits the data downstream.
   
   | Expression | Input | Before | After |
   |---|---|---|---|
   | `CEIL(long_v)` | `9007199254740993` | `1` | `9007199254740993` |
   | `FLOOR(double_v)` | `1.0E18` | `2147483647` | `1.0E18` |
   | `ROUND(decimal_v, 2)` | `12345678901234567890.987654321` | `12345678901234567000.00` | `12345678901234567890.99` |
   | `MOD(long_v, 2)` | `9007199254740993` | `0` | `1` |
   | `TRUNC(long_v)` | any | declared `DOUBLE`, returned a `Long` | declared type matches the value |
   
   This brings the implementation in line with the behaviour that `docs/en/transforms/sql-functions.md` already documents, which specifies that these functions return the data type of their argument:
   
   - `CEIL | CEILING (numeric) -> NUMERIC (same type, scale 0)`
   - `FLOOR(numeric) -> NUMERIC (same type, scale 0)`
   - `ROUND(numeric[, digitsInt]) -> NUMERIC (same type)`
   - `TRUNC | TRUNCATE(numeric[, digitsInt]) -> NUMERIC (same type)`
   - `MOD(dividendNumeric, divisorNumeric) -> type(divisorNumeric)`
   
   Changes:
   
   - `NumericFunction.ceil` / `floor` return `Number` instead of narrowing the result to `int`.
   - `NumericFunction.round` operates on the `BigDecimal` directly in its `BIGDECIMAL` branch instead of round-tripping through `BigDecimal.valueOf(v1.doubleValue())`, which dropped every digit a `double` cannot hold.
   - `NumericFunction.mod` uses exact `BigDecimal` arithmetic via a new `toBigDecimal` helper. This also fixes its divide-by-zero guard, which previously compared `doubleValue() == 0` and therefore missed divisors that underflow to `0.0`.
   - `ZetaSQLType` declares the argument's type for `CEIL` / `CEILING` / `FLOOR` / `TRUNC` / `TRUNCATE`, joining the existing `ABS` / `ROUND` group. This also resolves a declared-vs-actual type mismatch where `TRUNC` was declared `DOUBLE` but returned a `Long` at runtime.
   
   Two related defects are deliberately left out of scope, because each needs its own behavioural decision rather than being folded into this fix. I will raise them separately:
   
   1. `ABS(-2147483648)` returns `-2147483648`, since `Math.abs` cannot represent `-Integer.MIN_VALUE`. H2 raises an overflow error here, so converting a silent wrong answer into a hard job failure is a larger call than this PR should make.
   2. `ROUND(tinyint_col, -1)` does not round at all, because `Byte` is not handled in the type switch in `round()`. Adding it naively introduces a byte-overflow edge (`127` -> `130` -> `-126`), so it needs a widening decision first.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, and it is a change relative to released versions as well as to `dev`.
   
   1. `CEIL` / `CEILING`, `FLOOR` and `TRUNC` / `TRUNCATE` now return the data type of their argument rather than always `INT` (and rather than a declared `DOUBLE` for `TRUNC`). A `BIGINT` argument now yields a `BIGINT`, a `DOUBLE` argument a `DOUBLE`, and a `DECIMAL` argument a `DECIMAL`.
   2. `ROUND`, `TRUNC` / `TRUNCATE` and `MOD` now return exact results for `BIGINT` and `DECIMAL` arguments instead of values degraded by a `double` round-trip.
   
   Both are recorded in `docs/en/introduction/concepts/incompatible-changes.md` and its `docs/zh` mirror, with before/after tables and a migration guide. Pipelines that relied on the old narrowing should widen the target column, or wrap the call in `CAST(... AS INT)` to keep the previous output shape.
   
   No config option names, defaults or SPI contracts were changed.
   
   ### How was this patch tested?
   
   Added unit tests in `seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/sql/zeta/NumericFunctionTest.java`, which drive the real SQL engine and assert both the returned value and the declared output type:
   
   - `testCeilAndFloorKeepBigIntValue` — a `BIGINT` beyond `2^53` (`9007199254740993`) survives `CEIL` / `FLOOR` / `TRUNC` unchanged, and the declared type stays `LONG_TYPE`.
   - `testCeilAndFloorKeepDoubleType` — a `DOUBLE` of `1.0E18`, far outside the `int` range, keeps its value and `DOUBLE_TYPE`.
   - `testRoundAndTruncKeepDecimalPrecision` — a `DECIMAL(38, 9)` of `12345678901234567890.987654321` stays exact through `ROUND`, `TRUNC` and `CEIL`, with the declared type preserved.
   - `testModKeepsBigIntPrecision` — covers the `BIGINT` and `DECIMAL` cases, including the `type(divisorNumeric)` return-type rule.
   
   Two pre-existing assertions had codified the buggy narrowing (`assertEquals(2, ceil(1.2d))` in `SQLNumericFunctionsTest` and in the `...sql.zeta.functions.NumericFunctionTest` class) and were updated to the documented contract, with explicit `int` and `long` cases added alongside.
   
   Verified locally on JDK 11:
   
   ```bash
   ./mvnw spotless:apply
   ./mvnw -pl seatunnel-transforms-v2 -DskipTests verify
   ./mvnw -pl seatunnel-transforms-v2 test
   ```
   
   All pass: `Tests run: 1079, Failures: 0, Errors: 0, Skipped: 0`.
   
   Blast radius check: `ZetaSQLFunction` is the only caller of the changed methods and dispatches through `Object`, so widening the `ceil` / `floor` return type from `Integer` to `Number` is source- and binary-compatible for it. The remaining `CEIL(` / `FLOOR(` / `TRUNC(` matches in the repository are database-side SQL (`XuguCatalog`, a test resource config) and are unaffected.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [x] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Transform-V2] Fix silent value corruption in Zeta SQL numeric functions [seatunnel]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on PR #11697:
URL: https://github.com/apache/seatunnel/pull/11697#issuecomment-5230148122

   Thanks for the exceptionally thorough review — you clearly traced the runtime path rather than reading the diff in isolation, and the confirmation that `computeForValue`'s `targetType` is only consumed by `COALESCE`/`IFNULL` is the detail that makes the `TRUNC` type-contract argument airtight. Addressing your four points:
   
   **Issue 1 (CI never ran) — resolved.** GitHub Actions was indeed disabled on my fork. I've enabled it and re-pushed, and the workflows now execute: `Code style`, `License header`, `Build website`, `Dead links`, all four `unit-test` jobs (Java 8 and 11, Ubuntu and Windows) and `transform-v2-it-part-1` have passed, with the remaining connector integration jobs still running. No failures so far. For reference, locally on JDK 11 `./mvnw -pl seatunnel-transforms-v2 test` gives `Tests run: 1079, Failures: 0, Errors: 0, Skipped: 0`.
   
   **Issue 2 (comment placement) — fixed.** You're right that a comment between `case ROUND:` and `case CEIL:` reads as documenting only `ROUND`. I've moved it above the whole fallthrough group and reworded it to describe the group's shared contract:
   
   ```java
   // These functions all return the type of their first argument. CEIL/FLOOR/TRUNC are
   // documented that way as well; declaring INT/DOUBLE for them would truncate BIGINT
   // and DECIMAL results.
   case ZetaSQLFunction.ABS:
   case ZetaSQLFunction.DATEADD:
   case ZetaSQLFunction.TIMESTAMPADD:
   case ZetaSQLFunction.ROUND:
   case ZetaSQLFunction.CEIL:
   ...
   ```
   
   **Issue 3 (inverted bug direction in the description) — fixed, and you're correct.** I re-checked: the old `rightValue.doubleValue() == 0` guard had no false-negative path at all. Its only defect is the false positive you describe — a nonzero `DECIMAL` divisor below `Double.MIN_VALUE` underflows to `0.0` and gets rejected as "Mod by zero". I've corrected the PR description accordingly. The code (`rightBD.signum() == 0`) was already right.
   
   **Issue 4 (`DECIMAL` binary arithmetic) — agreed, and worth its own ticket.** Confirmed at `ZetaSQLFunction.executeBinaryExpr` (`ZetaSQLFunction.java:763-780`): `+`, `-`, `*` and `/` on `DECIMAL` operands all go through `BigDecimal.valueOf(leftValue.doubleValue())`, which is precisely the bug class this PR fixes for the function-call path. Notably `Modulo` in that same block already delegates to `NumericFunction.mod`, so it picks up this PR's fix for free — which leaves `+ - * /` as the sole remaining offenders and makes the inconsistency more visible. I'd rather not widen this PR's blast radius, so I'll open a follow-up issue for it. Same reasoning for the two I flagged originally (`ABS(Integer.MIN_VALUE)` overflow, and `Byte`/`TINYINT` missing from `round()`'s type switch).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Transform-V2] Fix silent value corruption in Zeta SQL numeric functions [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11697:
URL: https://github.com/apache/seatunnel/pull/11697


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Transform-V2] Fix silent value corruption in Zeta SQL numeric functions [seatunnel]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on PR #11697:
URL: https://github.com/apache/seatunnel/pull/11697#issuecomment-5231058650

   Pushed one more commit updating the `ceil`/`floor` assertions in `func_numeric.conf`.
   
   `c5` in that config is a `double`, and the Assert sink still expected `ceil(c5)` and `floor(c5)` to be `INT`. That was correct against the old hardcoded `BasicType.INT_TYPE`, but this PR makes those functions return the type of their argument, so they are now `DOUBLE` (`14.0` / `13.0`). The e2e config now asserts that.
   
   This is the only config in the repo that exercises `CEIL`/`FLOOR`, and it is the concrete demonstration of the behaviour change already recorded in `incompatible-changes.md`. The rest of the file is unchanged — I verified that `mod(c4, 5)` still yields `Integer` `3` (the divisor-type dispatch in `mod()` is untouched), `mod(c4, 5.4)` still yields `2.2`, and `round`/`trunc` on a `double` column keep the type they had before.
   
   For the record on the earlier CI run: both `transform-v2-it-part-2` legs were cancelled at the 150-minute job timeout rather than failing, and the `kafka-connector-it` / `all-connectors-it-7` failures are in modules this PR does not touch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Transform-V2] Fix silent value corruption in Zeta SQL numeric functions [seatunnel]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on PR #11697:
URL: https://github.com/apache/seatunnel/pull/11697#issuecomment-5231940865

   Thanks for the follow-up review, @DanielLeens. It looks like the body came through as a local file path (`.../drafts/11697.md`) rather than the review text — the file reference doesn't seem to have been expanded before posting, so I can't see the feedback.
   
   Could you re-post it? For context on what's changed since your first review: all four points from it are addressed, and I've since pushed one more commit (`ed97801`) updating the `ceil`/`floor` assertions in `func_numeric.conf`, which were still expecting `INT` for a `double` column.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix][Transform-V2] Fix silent value corruption in Zeta SQL numeric functions [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11697:
URL: https://github.com/apache/seatunnel/pull/11697#issuecomment-5233643940

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org