Kafka Streams and retention limits

[Dan Bradley <de...@gmail.com>]

I'm trying to understand the interactions between stateful processing in 
Kafka Streams and topic retention size restrictions. I'm fairly naive 
about both Kafka and Kafka Streams so I'm sure I have something wrong 
here, but my understanding is that, under stateful processing, local 
state in the application is made fault tolerant by the topic's commit 
log on the broker. That data, I assume, will count towards the topic's 
retention.bytes limit. If so, then what happens when I hit that limit? 
Let's say that the topic has a retention limit of 10mb and my stateful 
processor needs more than that to do its work. Does the processor fail? 
And what impact is there on retaining messages? Does the amount of 
messages stored in the topic trend toward 0 as the allocated space is 
used up by the processor?

Pointers to documentation that would explain this would also be helpful. 
I've tried to answer this question for myself but with no luck so far. 
Thank you very much.

Re: Kafka Streams and retention limits

[Bruno Cadonna <ca...@apache.org>]

Additionally, with KIP-698 
(https://cwiki.apache.org/confluence/x/7CnZCQ), we will add 
verifications of changelogs to find  misconfigurations and report them 
to the users.

Best,
Bruno

On 24.04.21 22:58, Matthias J. Sax wrote:
> The topics used by Kafka Streams to backup state stores, are called
> changelog topic, and they are configures with log compaction instead of
> retention, and thus, there is no size limit.
> 
> Cf https://kafka.apache.org/documentation/#compaction
> 
> 
> -Matthias
> 
> On 4/24/21 1:32 PM, Dan Bradley wrote:
>> I'm trying to understand the interactions between stateful processing in
>> Kafka Streams and topic retention size restrictions. I'm fairly naive
>> about both Kafka and Kafka Streams so I'm sure I have something wrong
>> here, but my understanding is that, under stateful processing, local
>> state in the application is made fault tolerant by the topic's commit
>> log on the broker. That data, I assume, will count towards the topic's
>> retention.bytes limit. If so, then what happens when I hit that limit?
>> Let's say that the topic has a retention limit of 10mb and my stateful
>> processor needs more than that to do its work. Does the processor fail?
>> And what impact is there on retaining messages? Does the amount of
>> messages stored in the topic trend toward 0 as the allocated space is
>> used up by the processor?
>>
>> Pointers to documentation that would explain this would also be helpful.
>> I've tried to answer this question for myself but with no luck so far.
>> Thank you very much.

Re: Kafka Streams and retention limits

["Matthias J. Sax" <mj...@apache.org>]

The topics used by Kafka Streams to backup state stores, are called
changelog topic, and they are configures with log compaction instead of
retention, and thus, there is no size limit.

Cf https://kafka.apache.org/documentation/#compaction


-Matthias

On 4/24/21 1:32 PM, Dan Bradley wrote:
> I'm trying to understand the interactions between stateful processing in
> Kafka Streams and topic retention size restrictions. I'm fairly naive
> about both Kafka and Kafka Streams so I'm sure I have something wrong
> here, but my understanding is that, under stateful processing, local
> state in the application is made fault tolerant by the topic's commit
> log on the broker. That data, I assume, will count towards the topic's
> retention.bytes limit. If so, then what happens when I hit that limit?
> Let's say that the topic has a retention limit of 10mb and my stateful
> processor needs more than that to do its work. Does the processor fail?
> And what impact is there on retaining messages? Does the amount of
> messages stored in the topic trend toward 0 as the allocated space is
> used up by the processor?
> 
> Pointers to documentation that would explain this would also be helpful.
> I've tried to answer this question for myself but with no luck so far.
> Thank you very much.