You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "nyingping (Jira)" <ji...@apache.org> on 2022/02/15 08:36:00 UTC

[jira] [Created] (SPARK-38214) No need to filter data when the sliding window length is not redundant

nyingping created SPARK-38214:
---------------------------------

             Summary: No need to filter data when the sliding window length is not redundant
                 Key: SPARK-38214
                 URL: https://issues.apache.org/jira/browse/SPARK-38214
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 3.2.1
            Reporter: nyingping


At present, the sliding window adopts the form of expand + filter, but in some cases, filter is not necessary.

Filter is required if the sliding window is irregular. When the window length is divided by the slide length the result is an integer (I believe this is also the case for most work scenarios in practice for sliding window), there is no need to filter, which can save calculation resources and improve performance.




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org