You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "daijy (via GitHub)" <gi...@apache.org> on 2026/04/08 06:57:51 UTC
[PR] [SPARK-56391] [SQL] Support unwrap string type to date type in UnwrapCastInBinaryComparison [spark]
daijy opened a new pull request, #55257:
URL: https://github.com/apache/spark/pull/55257
### What changes were proposed in this pull request?
Improve the existing UnwrapCastInBinaryComparison to unwrap string to date, ie. cast(date_key_string as date)=date_literal_constant => date_key_string=cast(date_literal_constant, string)
### Why are the changes needed?
Spark fail to push date_key_string=date_literal_constant condition to iceberg table, this works fine for parquet table (date_key is a string partition column):
```
explain select date_key, count(*) from roku.fact_device_log_amoeba_events_hourly where date_key=date_add(current_date, -10) group by date_key;
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[date_key#25], functions=[count(1)])
+- Exchange hashpartitioning(date_key#25, 200), ENSURE_REQUIREMENTS, [plan_id=16]
+- HashAggregate(keys=[date_key#25], functions=[partial_count(1)])
+- Filter (cast(date_key#25 as date) = 2026-03-28)
+- BatchScan spark_catalog.roku.fact_device_log_amoeba_events_hourly[date_key#25] spark_catalog.roku.fact_device_log_amoeba_events_hourly (branch=null) [filters=date_key IS NOT NULL, groupedBy=] RuntimeFilters: []
```
The reason is Spark add a cast on the left side to cast string to date. The cast expression pushed to iceberg data source which is not able to process. A workaround is to add cast on right side, ie. cast date to string.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit test
### Was this patch authored or co-authored using generative AI tooling?
claude-4.5-sonnet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56391] [SQL] Support unwrap string type to date type in UnwrapCastInBinaryComparison [spark]
Posted by "daijy (via GitHub)" <gi...@apache.org>.
daijy closed pull request #55257: [SPARK-56391] [SQL] Support unwrap string type to date type in UnwrapCastInBinaryComparison
URL: https://github.com/apache/spark/pull/55257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56391] [SQL] Support unwrap string type to date type in UnwrapCastInBinaryComparison [spark]
Posted by "daijy (via GitHub)" <gi...@apache.org>.
daijy commented on PR #55257:
URL: https://github.com/apache/spark/pull/55257#issuecomment-4347662136
Actually there's one issue introduced in the patch. to_date could also truncate a string, so the filter condition is not equivalent after the rule. Here is the reproduction:
```
CREATE EXTERNAL TABLE jidai_dev.to_date_test (
device_first_linked STRING
)
USING parquet
LOCATION 'gs://roku-dea-dev/dev/jidai/to_date_test';
insert into jidai_dev.to_date_test values ('2026-04-16 08:00:00');
SELECT device_first_linked FROM jidai_dev.to_date_test WHERE to_date(device_first_linked) = '2026-04-16'; -- return empty
```
After the UnwrapCastInBinaryComparison:
```
to_date(device_first_linked) = date'2026-04-16'
=>
device_first_linked = cast(date'2026-04-16' as string)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org