Kafka Streams Hang While Committing

[Sicheng Liu <lz...@gmail.com>]

Hi Kafka Community,

I have a question regarding Kafka Streams. Currently we set our commit
interval to 5 mins and buffer size to be 1GB. We observe that every 5
minutes, there is a spike of network out, which is expected. However, we
also found that the record process rate dropped to 1/4 during committing.
Is this expected? If not, how to address this issue?

Thanks,
Sicheng

Re: Kafka Streams Hang While Committing

["Matthias J. Sax" <ma...@confluent.io>]

Well. When a commit a triggered, Streams need to flush all caches and
flush all pending write of the producers. And as this happens on the
same thread that does processing, there won't be any processing of new
data until the commit is finished.

So I guess, it is expected.

-Matthias



On 10/24/17 1:17 PM, Sicheng Liu wrote:
> Hi Kafka Community,
> 
> I have a question regarding Kafka Streams. Currently we set our commit
> interval to 5 mins and buffer size to be 1GB. We observe that every 5
> minutes, there is a spike of network out, which is expected. However, we
> also found that the record process rate dropped to 1/4 during committing.
> Is this expected? If not, how to address this issue?
> 
> Thanks,
> Sicheng
>