You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/10/04 17:59:16 UTC
Re: [PR] fix: preserve Spark evaluation for next_day and levenshtein [datafusion-comet]
sunchao commented on PR #5972:
URL: https://github.com/apache/datafusion-comet/pull/5972#issuecomment-5982822631
Addressed the review in the updated branch, stacked on #5533:
- Enrolled `NextDay` in the shared policy; removed the duplicate traversal, retained the early exit/configuration and main's shared aggregate restoration.
- Added valid-weekday admission below LIMIT, eager parent/sort admission, and native unfiltered MAX controls. Serializer and deferred-reference analysis share their eager-input rule; aggregate fallback is scoped to FILTER and First/Last.
- Nullable three-argument `levenshtein` now stays in Spark for FALLBACK, CODEGEN_ONLY and NO_CODEGEN, including containing JVM-dispatched expressions. Safe forms remain native.
- Generalized whole-expression dispatch, shared the interpreted-mode helper, restored the raw-bytes explanation, and extended the existing compatibility section.
The deferred-projection analysis remains in this follow-up to the shared policy rather than introducing another open dependent PR. Its scope is covered by six focused planner tests and the nine-test shared mask suite, in addition to nine next_day and eight levenshtein SQL cases. All 32 focused tests pass on Spark 4.1.3 across the current-source runs; other profiles and full Spark SQL suites are requested in CI.
The Spark JIRA filing is still outstanding: the available Jira connector is not authenticated. I checked current Spark master and the same eval/doGenCode discrepancy remains. Here is a ready-to-file report so that limitation is explicit:
**Title:** Nullable levenshtein threshold returns different results in interpreted and generated projections
Run with Comet disabled, using stored input to avoid constant folding:
```sql
CREATE TABLE levenshtein_null_threshold (s STRING, t STRING, threshold INT) USING parquet;
INSERT INTO levenshtein_null_threshold VALUES ('same', 'same', NULL), ('a', 'b', NULL);
SET spark.sql.codegen.wholeStage=false;
SET spark.sql.codegen.factoryMode=NO_CODEGEN;
SELECT levenshtein(s, t, threshold) FROM levenshtein_null_threshold;
SET spark.sql.codegen.factoryMode=CODEGEN_ONLY;
SELECT levenshtein(s, t, threshold) FROM levenshtein_null_threshold;
```
`Levenshtein.eval` passes `Some(null)` through `asInstanceOf[Int]`, producing a zero threshold; generated evaluation checks threshold nullness and returns NULL. Expected: both paths return NULL. The default FALLBACK mode can encounter either projection implementation after a compilation failure. The Comet workaround must remain until upstream behavior is consistent across supported Spark versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org