You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2021/07/15 16:48:00 UTC
[jira] [Resolved] (SPARK-36157) TimeWindow expression: apply filter
before project
[ https://issues.apache.org/jira/browse/SPARK-36157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
L. C. Hsieh resolved SPARK-36157.
---------------------------------
Fix Version/s: 3.3.0
Resolution: Fixed
Issue resolved by pull request 33367
[https://github.com/apache/spark/pull/33367]
> TimeWindow expression: apply filter before project
> --------------------------------------------------
>
> Key: SPARK-36157
> URL: https://issues.apache.org/jira/browse/SPARK-36157
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Structured Streaming
> Affects Versions: 3.3.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Minor
> Fix For: 3.3.0
>
>
> https://github.com/apache/spark/blob/4dfd266b27fea6954593c6b9e3a2819b290f0aec/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L3901-L3906
> In the case of tumbling window, we apply project and then filter, while filter is not dependent to project. We can just swap two operator to ensure less rows would be projected if there're some rows being filtered out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org