[jira] [Created] (KAFKA-13811) Investigate sliding windows performance

["Leah Thomas (Jira)" <ji...@apache.org>]

Leah Thomas created KAFKA-13811:
-----------------------------------

             Summary: Investigate sliding windows performance
                 Key: KAFKA-13811
                 URL: https://issues.apache.org/jira/browse/KAFKA-13811
             Project: Kafka
          Issue Type: Task
          Components: streams
            Reporter: Leah Thomas


We recently fixed a bug in sliding windows so that a grace period of 0ms is properly calculated, see https://issues.apache.org/jira/browse/KAFKA-13739. Before this patch, sliding windows with a grace period of 0ms would just skip all records so nothing would get put into the store.

When we ran benchmarks for the 3.2 release we saw a significant drop in performance for sliding windows on both the 3.2 and trunk branches, see the `sliding windows` results [here|[http://kstreams-benchmark-results.s3-website-us-west-2.amazonaws.com/summaries/process-cumulative-rate/graph.html].] These benchmarks use a sliding window with a 0ms grace period, which means until now we weren't sending any values to the state store when running these benchmarks.

I ran benchmarks on the [commit|https://github.com/apache/kafka/commit/430f9c99012d1585aa544d4dadf449963296c1fd] before KAFKA-13739 and changed the grace period to 2 seconds to see if the bug fix changed anything. The performance was still low for [this run|[http://kstreams-benchmark-results.s3-website-us-west-2.amazonaws.com/experiments/sliding1min-3-5-3-3-0-430f9c9901-leah-20220408084241-streamsbench/].]

Given this, it seems like the performance for sliding windows has always been low but we didn't realize it because the bug fixed in KAFKA-13739 was present in the benchmarks we were running. We should investigate why the current algorithm is slow and see if improvements can be made



--
This message was sent by Atlassian Jira
(v8.20.1#820001)