kafka-streams: interaction between max.poll.records and window expiration ?

[Mathieu D <ma...@gmail.com>]

Hello there,

One of our input topics does not have so much traffic.
Divided by the number of partitions, and given the default 'max.poll.records'
setting (being 1000 if I understand the doc correctly), it could happen
that fetching 1000 records at once, the event timestamps between the first
and last record in the "batch" could be larger than some windows in my
topology.

Could this have any impact on window expiration ?

Thanks
Mathieu

Re: kafka-streams: interaction between max.poll.records and window expiration ?

[John Roesler <vv...@apache.org>]

Hi Mathieu,

I don’t think there would be any problem. Note that window expiry is computed against an internal clock called “stream time”, which is the max timestamp yet observed. This time is advanced per each record when that record is processed. There is a separate clock for each partition, so they will not affect each other.

I hope this helps,
John

On Sun, Dec 20, 2020, at 08:22, Mathieu D wrote:
> Hello there,
> 
> One of our input topics does not have so much traffic.
> Divided by the number of partitions, and given the default 'max.poll.records'
> setting (being 1000 if I understand the doc correctly), it could happen
> that fetching 1000 records at once, the event timestamps between the first
> and last record in the "batch" could be larger than some windows in my
> topology.
> 
> Could this have any impact on window expiration ?
> 
> Thanks
> Mathieu
>