You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "EshwarCVS (via GitHub)" <gi...@apache.org> on 2026/05/07 05:05:56 UTC
[PR] fix(spark): ANSI-aware custom nullability for `make_interval` [datafusion]
EshwarCVS opened a new pull request, #22057:
URL: https://github.com/apache/datafusion/pull/22057
## Which issue does this PR close?
- Closes #19155
## Rationale for this change
Spark's `make_interval` nullability follows this rule (from Spark source):
```scala
nullable = if (failOnError) children.exists(_.nullable) else true
Where failOnError maps to DataFusion's enable_ansi_mode. The previous
default implementation always reports nullable = true, which is correct
for non-ANSI mode (overflow silently returns NULL) but too pessimistic for
ANSI mode (overflow throws, so output is only nullable when inputs are).
```
Previous attempts ([#19248](https://github.com/EshwarCVS/datafusion/issues/19248), [#19606](https://github.com/EshwarCVS/datafusion/issues/19606)) tried to derive nullability from input
fields inside return_field_from_args, but ReturnFieldArgs carries no
ConfigOptions, so ANSI mode was inaccessible.
### What changes are included in this PR?
Add ansi_mode: bool field to SparkMakeInterval, populated via a new
new_with_config(config) constructor.
Implement with_updated_config so the planner keeps the UDF in sync with
session config (same pattern as SparkCast).
Implement return_field_from_args with Spark's exact rule:
nullary call → nullable = false (always returns zero interval)
ANSI mode on → nullable = any input field is nullable
ANSI mode off (default) → nullable = true
In ANSI mode, make_interval_kernel now returns exec_err! on arithmetic
overflow instead of silently appending NULL.
### Are these changes tested?
Yes. Six new unit tests cover all branches:
```
return_field_nullary_is_not_nullable — zero-arg call is never nullable
return_field_non_ansi_always_nullable — non-ANSI: always nullable even with non-null inputs
return_field_ansi_mode_not_nullable_when_inputs_not_null — ANSI + non-null inputs → not nullable
return_field_ansi_mode_nullable_when_any_input_nullable — ANSI + nullable input → nullable
ansi_mode_overflow_returns_error — ANSI mode overflow is an error, not NULL
non_ansi_overflow_returns_null — non-ANSI overflow is still NULL (regression guard)
All 14 tests (8 existing + 6 new) pass.
```
### Are there any user-facing changes?
Non-ANSI mode (default): no behavior change.
ANSI mode: make_interval with non-nullable inputs now correctly
reports a non-nullable output field, and arithmetic overflow raises an
error instead of silently returning NULL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): ANSI-aware custom nullability for `make_interval` [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #22057:
URL: https://github.com/apache/datafusion/pull/22057#issuecomment-4399416884
I am not in favor of this PR. For the default/non-ANSI mode, the current behavior already matches Spark: `make_interval` is nullable and overflow returns NULL.
The ANSI-mode part needs a broader design discussion because it changes core UDF return-field API and still does not prove the full SQL/planning path works correctly.
Given that, I think we should close this PR for now instead of making this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): ANSI-aware custom nullability for `make_interval` [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22057: fix(spark): ANSI-aware custom nullability for `make_interval`
URL: https://github.com/apache/datafusion/pull/22057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): ANSI-aware custom nullability for `make_interval` [datafusion]
Posted by "EshwarCVS (via GitHub)" <gi...@apache.org>.
EshwarCVS commented on PR #22057:
URL: https://github.com/apache/datafusion/pull/22057#issuecomment-4398674385
> @EshwarCVS can you take a look at this https://github.com/apache/datafusion/issues/19155#issuecomment-3694002373
updated
Lmk if it needs any update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): ANSI-aware custom nullability for `make_interval` [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22057:
URL: https://github.com/apache/datafusion/pull/22057#issuecomment-4899466397
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(spark): ANSI-aware custom nullability for `make_interval` [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #22057:
URL: https://github.com/apache/datafusion/pull/22057#issuecomment-4395446436
@EshwarCVS can you take a look at this https://github.com/apache/datafusion/issues/19155#issuecomment-3694002373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org