Merging create/delete updates for Kafka Streams aggregations

[Thilo-Alexander Ginkel <th...@ginkel.com>]

Hello everyone,

we are using the Kafka Streams DSL (v2.2.1) to perform aggregations on a
topic using org.apache.kafka.streams.kstream.KGroupedTable#aggregate. ATM
we are seeing one update being published when the subtractor is being
called and another one when the adder is called.

I was under the impression that there is some setting that allows
folding/merging these updates for most cases.

As I can't seem to find this in the docs, can someone please help me out
with a pointer?

Thanks & kind regards,
Thilo

Re: Merging create/delete updates for Kafka Streams aggregations

[Bruno Cadonna <br...@confluent.io>]

Hi Thilo,

You can influence the rate of updates of aggregations by configuring
the size of the record caches with `cache.max.bytes.buffering`.

Details can be found here:
https://kafka.apache.org/22/documentation/streams/developer-guide/dsl-api.html#aggregating
https://kafka.apache.org/22/documentation/streams/developer-guide/memory-mgmt.html#streams-developer-guide-memory-management-record-cache

If you aggregate over windows, you should have a look at `suppress()`.

Details can be found here:
 https://kafka.apache.org/22/documentation/streams/developer-guide/dsl-api.html#window-final-results

Best,
Bruno

On Fri, Nov 15, 2019 at 12:02 PM Thilo-Alexander Ginkel
<th...@ginkel.com> wrote:
>
> Hello everyone,
>
> we are using the Kafka Streams DSL (v2.2.1) to perform aggregations on a
> topic using org.apache.kafka.streams.kstream.KGroupedTable#aggregate. ATM
> we are seeing one update being published when the subtractor is being
> called and another one when the adder is called.
>
> I was under the impression that there is some setting that allows
> folding/merging these updates for most cases.
>
> As I can't seem to find this in the docs, can someone please help me out
> with a pointer?
>
> Thanks & kind regards,
> Thilo