You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2022/02/18 01:18:00 UTC

[jira] [Resolved] (SPARK-38214) No need to filter windows when windowDuration is multiple of slideDuration

     [ https://issues.apache.org/jira/browse/SPARK-38214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-38214.
----------------------------------
    Fix Version/s: 3.3.0
       Resolution: Fixed

Issue resolved by pull request 35526
[https://github.com/apache/spark/pull/35526]

> No need to filter windows when windowDuration is multiple of slideDuration
> --------------------------------------------------------------------------
>
>                 Key: SPARK-38214
>                 URL: https://issues.apache.org/jira/browse/SPARK-38214
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.2.1
>            Reporter: nyingping
>            Assignee: nyingping
>            Priority: Minor
>             Fix For: 3.3.0
>
>
> At present, the sliding window adopts the form of expand + filter, but in some cases, filter is not necessary.
> Filter is required if the sliding window is irregular. When the window length is divided by the slide length the result is an integer (I believe this is also the case for most work scenarios in practice for sliding window), there is no need to filter, which can save calculation resources and improve performance.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org