You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "omerhadari (via GitHub)" <gi...@apache.org> on 2025/02/18 18:18:03 UTC
[I] Date Cast expressions are turned into incorrect pushdown predicates in DataFusion [iceberg-rust]
omerhadari opened a new issue, #976:
URL: https://github.com/apache/iceberg-rust/issues/976
When turning expressions of type `Cast` that cast to `Date`, expressions are truncated to the nearest day. However, when creating predicates in `expr_to_predicate.rs` the expression is simply extracted as is. This can create incorrect predicates and filter out files with relevant data.
For example currently `SELECT * FROM table WHERE col >= CAST('2025-01-01T11:11:11' AS DATE)` creates the predicate `col >= '2025-01-01T11:11:11'` which excludes data files where `col >= 2025-01-01 AND col < 2025-01-01T11:11:11`, which is a mistake.
DataFusion should simplify such casts to Literal expressions, but it cannot be assume that no CAST expressions reach this part (for example, simplifiers can be turned off in DataFusion).
This issue was originally raised as part of #933 but is distinct and warrants a separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Date Cast expressions are turned into incorrect pushdown predicates in DataFusion [iceberg-rust]
Posted by "omerhadari (via GitHub)" <gi...@apache.org>.
omerhadari commented on issue #976:
URL: https://github.com/apache/iceberg-rust/issues/976#issuecomment-2666524945
opening a PR soon :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Date Cast expressions are turned into incorrect pushdown predicates in DataFusion [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #976:
URL: https://github.com/apache/iceberg-rust/issues/976#issuecomment-2667425504
Thanks @omerhadari , #977 merged and we should close this issue now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Date Cast expressions are turned into incorrect pushdown predicates in DataFusion [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 closed issue #976: Date Cast expressions are turned into incorrect pushdown predicates in DataFusion
URL: https://github.com/apache/iceberg-rust/issues/976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org