You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xiedeyantu (via GitHub)" <gi...@apache.org> on 2026/04/02 13:39:08 UTC
[PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
xiedeyantu opened a new pull request, #21319:
URL: https://github.com/apache/datafusion/pull/21319
## Which issue does this PR close?
- Closes #21318 .
## Rationale for this change
DataFusion was silently wrapping on integer arithmetic overflow instead of returning an error. For example:
```
SELECT CAST(32767 AS SMALLINT) + CAST(2 AS SMALLINT);
-- Before: -32767 (silent wrapping)
-- After: Error: Arithmetic overflow: Overflow happened on: 32767 + 2
```
This behavior is incorrect. DuckDB and PostgreSQL both return an error on integer overflow, which is the expected behavior for SQL engines.
## What changes are included in this PR?
- `BinaryExpr::new (binary.rs)`: Change the default value of `fail_on_overflow` from `false` to `true`. This causes `+`, `-`, and `*` on integer types to use Arrow's checked kernels instead of the wrapping variants, returning an error on overflow.
- Update snapshot assertions in `dynamic_filters.rs` and `physical_planner.rs` to reflect the new default.
- Update `explain_analyze.slt` to use slt:ignore for `output_bytes` on integer arithmetic projections, since Arrow's checked kernels may allocate a validity buffer, slightly changing the reported byte size.
- Add sqllogictest cases in `operator.slt` covering overflow for `TINYINT`, `SMALLINT`, `INT`, and `BIGINT` with `+`, `-`, and `*`.
## Are these changes tested?
Yes.
- New query error tests in `operator.slt`verify that overflow on all integer widths (`Int8`, `Int16`, `Int32`, `Int64`) returns an error for addition, subtraction, and multiplication.
- Existing tests `test_add_with_overflow`, `test_subtract_with_overflow`, and `test_mul_with_overflow`in `binary.rs`continue to pass (they already used `with_fail_on_overflow(true)`explicitly).
## Are there any user-facing changes?
Yes. Integer arithmetic (`+`, `-`, `*`) on types `TINYINT`, `SMALLINT`, `INT`, and `BIGINT` now returns an error on overflow instead of silently wrapping. This aligns DataFusion's behavior with DuckDB and PostgreSQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4183881387
The datafusion-testing repository should be updated to align with this change. I have forked the datafusion-testing project on my GitHub and submitted a https://github.com/apache/datafusion-testing/pull/18. I'm not sure about the merging process. The current step is to update the submodule using the datafusion-testing [PR](https://github.com/apache/datafusion-testing/pull/18).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4194514008
I am trying to figure out if this is an theroetical concern or if it is an error that someone has actually reported?
I just worry it will introduce churn / failures for people's whose queries are currently running fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4192850272
@alamb Could you please take a look and let me know if this issue needs to be fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4227022024
If no one is complaining about it, I would personally say "no"
On Fri, Apr 10, 2026 at 12:13 PM Zhen Chen ***@***.***> wrote:
> *xiedeyantu* left a comment (apache/datafusion#21319)
> <https://github.com/apache/datafusion/pull/21319#issuecomment-4225144725>
>
> @alamb <https://github.com/alamb> May I ask if we need to fix this?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/datafusion/pull/21319#issuecomment-4225144725>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADXZMJU3ZMUNCJ66YTKSID4VEMT7AVCNFSM6AAAAACXKTNXZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEMRVGE2DINZSGU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4229296862
@alamb Of course, I'll leave this question on hold for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4195427243
> I am trying to figure out if this is an theroetical concern or if it is an error that someone has actually reported?
>
> I just worry it will introduce churn / failures for people's whose queries are currently running fine.
In any case, I believe this must be an error. However, different query engines may exhibit their own distinct behaviors. If you feel that this issue does not require a fix, we can certainly leave things as they are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21319:
URL: https://github.com/apache/datafusion/pull/21319#issuecomment-4225144725
@alamb May I ask if we need to fix this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: SMALLINT addition overflow should error instead of silently wrapping [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu closed pull request #21319: fix: SMALLINT addition overflow should error instead of silently wrapping
URL: https://github.com/apache/datafusion/pull/21319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org