You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "ShreyeshArangath (via GitHub)" <gi...@apache.org> on 2026/08/19 18:09:14 UTC
[I] make_date raises an execution error on NULL year/month/day instead of returning NULL [auron]
ShreyeshArangath opened a new issue, #2474:
URL: https://github.com/apache/auron/issues/2474
### Describe the bug
Since `make_date` was offloaded to the native engine (#2240 / #2241), `make_date(y, m, d)` fails the task whenever a component is NULL, instead of producing NULL.
Spark's `MakeDate` is `NullIntolerant`, and with `spark.sql.ansi.enabled=false` (the default) it returns NULL for NULL components and for out-of-range components. Under Auron the task dies:
```
task panics: Execution error: output_with_sender[Project]: output() returns error:
Execution error: Unable to parse date from null/empty value
```
This is not limited to literals — a plain column containing nulls fails too:
```sql
create table t2 using parquet as
select 2025 as y, 3 as m, 1 as d union all select 2024, cast(null as int), 2;
select make_date(y, m, d) from t2;
-- Execution error: Unable to parse date from 2024, 0, 2
```
Vanilla Spark returns `[2025-03-01, null]`.
### Root cause
The offload maps Spark's `MakeDate` directly onto DataFusion's stock `make_date` kernel, which does not implement Spark's null or range semantics:
1. **Typed nulls fall through.** The kernel early-returns NULL only for the untyped `ScalarValue::Null`. Auron emits a *typed* null literal (the logged plan shows `make_date(NULL, 7, 15)`, an `Int32(None)` scalar), which reaches `scalar_value_fn` and hits `exec_err!("Unable to parse date from null/empty value")`.
2. **The columnar path ignores the Arrow null mask.** `to_primitive_array_fn` calls `years.value(i)` / `months.value(i)` / `days.value(i)` with no validity check, so a null slot reads the raw buffer (`0`) and `NaiveDate::from_ymd_opt(2024, 0, 2)` returns `None` → `exec_err!("Unable to parse date from 2024, 0, 2")`. The output builder only ever calls `append_value`, so it cannot emit nulls at all.
3. **Out-of-range components error rather than nulling.** Spark non-ANSI catches `DateTimeException` and returns NULL.
4. **`failOnError` is dropped.** `MakeDate.failOnError` is not carried into the proto, so ANSI mode also cannot raise Spark's `DateTimeException("Invalid value for Month")`.
Sites on master:
- `spark-extension/src/main/scala/org/apache/spark/sql/auron/NativeConverters.scala:839` — `case e: MakeDate => buildScalarFunction(pb.ScalarFunction.MakeDate, e.children, e.dataType)`, unconditional, `failOnError` discarded.
- `native-engine/auron-planner/src/planner.rs:1392` — `ScalarFunction::MakeDate => f::datetime::make_date()`.
- DataFusion `datafusion/functions/src/datetime/make_date.rs`, `invoke_with_args` and `make_date_inner`.
Spark's own `DateExpressionsSuite` test "creating values of DateType via make_date" catches this — its first three assertions are exactly the NULL-component cases.
### Suggested fix
Add a Spark-semantics `Spark_MakeDate` ext function in `datafusion-ext-functions` (following the `Spark_ArrayReverse` precedent) that respects the Arrow null mask, propagates NULL for any null component, and returns NULL rather than erroring for unrepresentable dates. Plumb `failOnError` through the proto so ANSI mode raises `DateTimeException` with Spark's message.
A cheap stopgap is to guard the converter so `MakeDate` only converts when all three children are non-nullable and `failOnError` is false, falling back to Spark otherwise.
### Environment
Reproduced on Spark 3.1.1 / JDK 11 against a fork carrying #2241 verbatim. Master carries the identical mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make_date raises an execution error on NULL year/month/day instead of returning NULL [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 closed issue #2474: make_date raises an execution error on NULL year/month/day instead of returning NULL
URL: https://github.com/apache/auron/issues/2474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make_date raises an execution error on NULL year/month/day instead of returning NULL [auron]
Posted by "ShreyeshArangath (via GitHub)" <gi...@apache.org>.
ShreyeshArangath commented on issue #2474:
URL: https://github.com/apache/auron/issues/2474#issuecomment-5404297021
Thanks for picking it up @Sigma-Ma! I'll take a look at this soon :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make_date raises an execution error on NULL year/month/day instead of returning NULL [auron]
Posted by "Sigma-Ma (via GitHub)" <gi...@apache.org>.
Sigma-Ma commented on issue #2474:
URL: https://github.com/apache/auron/issues/2474#issuecomment-5395995213
Hi @ShreyeshArangath, as the issue remained open and unassigned, I went ahead and prepared a fix in #2479 .
The change replaces the stock DataFusion `make_date` mapping with a Spark-specific `Spark_MakeDate` extension function. It forwards `failOnError`, propagates scalar and columnar NULLs, returns NULL for invalid inputs in non-ANSI mode, and preserves ANSI-mode failure behavior.
I also added Rust and Spark regression coverage for typed NULLs, columnar NULLs, invalid dates, and both ANSI modes.
Please take a look when you have time. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] make_date raises an execution error on NULL year/month/day instead of returning NULL [auron]
Posted by "Sigma-Ma (via GitHub)" <gi...@apache.org>.
Sigma-Ma commented on issue #2474:
URL: https://github.com/apache/auron/issues/2474#issuecomment-5365805920
Hi @ShreyeshArangath, I'd like to work on this issue. Could you please assign it to me?
I plan to replace the stock DataFusion `make_date` mapping with a Spark-specific extension function, following the existing `Spark_*` pattern. It will:
- propagate typed and columnar NULLs;
- return NULL for invalid dates when `failOnError` is false;
- preserve ANSI-mode failure behavior by carrying `failOnError` through the existing extension-function arguments.
I'll also add Rust coverage for scalar/columnar NULLs and invalid dates, together with Spark regression tests for both ANSI modes. Please let me know if you would prefer the conservative converter fallback instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org