You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/02/15 09:02:00 UTC

[jira] [Assigned] (SPARK-38214) No need to filter data when the sliding window length is not redundant

     [ https://issues.apache.org/jira/browse/SPARK-38214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-38214:
------------------------------------

    Assignee: Apache Spark

> No need to filter data when the sliding window length is not redundant
> ----------------------------------------------------------------------
>
>                 Key: SPARK-38214
>                 URL: https://issues.apache.org/jira/browse/SPARK-38214
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.2.1
>            Reporter: nyingping
>            Assignee: Apache Spark
>            Priority: Minor
>
> At present, the sliding window adopts the form of expand + filter, but in some cases, filter is not necessary.
> Filter is required if the sliding window is irregular. When the window length is divided by the slide length the result is an integer (I believe this is also the case for most work scenarios in practice for sliding window), there is no need to filter, which can save calculation resources and improve performance.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org