You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2021/07/15 11:32:00 UTC

[jira] [Created] (SPARK-36157) TimeWindow expression: apply filter before project

Jungtaek Lim created SPARK-36157:
------------------------------------

             Summary: TimeWindow expression: apply filter before project
                 Key: SPARK-36157
                 URL: https://issues.apache.org/jira/browse/SPARK-36157
             Project: Spark
          Issue Type: Improvement
          Components: SQL, Structured Streaming
    Affects Versions: 3.3.0
            Reporter: Jungtaek Lim


https://github.com/apache/spark/blob/4dfd266b27fea6954593c6b9e3a2819b290f0aec/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L3901-L3906

In the case of tumbling window, we apply project and then filter, while filter is not dependent to project. We can just swap two operator to ensure less rows would be projected if there're some rows being filtered out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org