[jira] [Resolved] (KAFKA-18018) Consumption degrading when using DefaultStatePersister for 1 million records of size 1KB

["Abhinav Dixit (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-18018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Abhinav Dixit resolved KAFKA-18018.
-----------------------------------
    Resolution: Later

> Consumption degrading when using DefaultStatePersister for 1 million records of size 1KB
> ----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-18018
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18018
>             Project: Kafka
>          Issue Type: Sub-task
>            Reporter: Abhinav Dixit
>            Assignee: Abhinav Dixit
>            Priority: Major
>
> I was running some performance tests and I couldn't consume 1,000,000 records using 10 share consumers. I've narrowed down the issue to {{DefaultStatePersister}} . When I run the same tests using {{{}NoOpShareStatePersister{}}}, then I don't see the reliability issue.
> Steps to reproduce (could be maybe reproduced by other ways as well, but this is how I noticed it consistently) - # Create a topic with a single partition
>  # Produce 1000 records into the topic and consume it with the help of 10 share consumers.
>  # Again produce 1000 records into the topic and consume it with the help of 10 share consumers.
>  # Produce 1,000,000 records into the topic and consume it with the help of 10 share consumers.
> You'll see the records not getting consumed even within 5 minutes (It shouldn't take more than 5-6 seconds to consume 1,000,000 records with 10 share consumers, based on my past experience)
> PS - I haven't noticed any issues using console share consumers, so probably an issue with scale



--
This message was sent by Atlassian Jira
(v8.20.10#820010)