You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Holden Karau (Jira)" <ji...@apache.org> on 2024/04/01 17:50:00 UTC
[jira] [Commented] (SPARK-47672) Avoid double evaluation of non-trivial projected elements from filter pushdown
[ https://issues.apache.org/jira/browse/SPARK-47672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832916#comment-17832916 ]
Holden Karau commented on SPARK-47672:
--------------------------------------
I'm going to work on this :)
> Avoid double evaluation of non-trivial projected elements from filter pushdown
> ------------------------------------------------------------------------------
>
> Key: SPARK-47672
> URL: https://issues.apache.org/jira/browse/SPARK-47672
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.1
> Reporter: Holden Karau
> Priority: Major
>
> Repro here [https://gist.github.com/holdenk/0f9660bcbd9e63aaff904f15d3439db1]
>
> You can work around this by setting an expensive UDF to non-deterministic but that's not ideal and won't fix expensive internal operations (like string matching).
>
> Instead when we go to bubble up a filter, if we should not move a filter up above a projection of what we are filtering on.
>
> https://issues.apache.org/jira/browse/SPARK-40045 partially fixed some of this by (roughly) ordering filter expressions by cost so that we're not evaluating more than ~2x (e.g. in old behavior bubbled up filter could become the first elem of the filter and then the cheap null checks would go away and we'd have expensive compute on everything not just filtered data), but we should "trust" the users projection + later use of that projection to indicate that a UDF is expensive and we should only evaluate it once inside of the projection and filter after.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org