You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ulrik (Jira)" <ji...@apache.org> on 2022/02/02 12:54:00 UTC
[jira] [Updated] (KAFKA-13638) Slow KTable update when forwarding multiple values from transformer
[ https://issues.apache.org/jira/browse/KAFKA-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ulrik updated KAFKA-13638:
--------------------------
Description:
I have a topology where I stream messages from an input topic, transform the message to multiple messages (via context.forward), and then store those messages in a KTable.
Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my tests takes significantly longer time to run.
I have attached a test class to demonstrate my scenario. When running this test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following numbers:
*Version 2.8.1*
* one input message and one output message: 541 ms
* 8 input message and 30 output message per input message (240 output messages in total): 919 ms
*Version 3.1.0*
* one input message and one output message: 908 ms
* 8 input message and 30 output message per input message (240 output messages in total): 6 sec 94 ms
Even when the transformer just transforms and forwards one input message to one output message, the test takes approx. 400 ms longer to run.
When transforming 8 input messages to 240 output messages it takes approx 5 seconds longer.
was:
I have a topology where I stream messages from an input topic, transform the message to multiple messages (via context.forward), and then store those messages in a KTable.
Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my tests takes significantly longer time to run.
I have attached a test class to demonstrate my scenario. When running this test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following numbers:
*Version 2.8.1*
* one input message and one output message: 541 ms
* 8 input message and 30 output message per input message (240 output messages in total): 919 ms
*Version 3.1.0*
* one input message and one output message: 908 ms
* 8 input message and 30 output message per input message (240 output messages in total): 6 sec 94 ms
Even when the transformer just transforms and forwards one input message to one output message, the test takes approx. 400 ms longer to run.
When transforming 8 input messages to 240 output messages it takes approx 5 seconds longer.
> Slow KTable update when forwarding multiple values from transformer
> -------------------------------------------------------------------
>
> Key: KAFKA-13638
> URL: https://issues.apache.org/jira/browse/KAFKA-13638
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.1.0, 3.0.0
> Reporter: Ulrik
> Priority: Major
> Attachments: KafkaTest.java
>
>
> I have a topology where I stream messages from an input topic, transform the message to multiple messages (via context.forward), and then store those messages in a KTable.
> Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my tests takes significantly longer time to run.
>
> I have attached a test class to demonstrate my scenario. When running this test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following numbers:
>
> *Version 2.8.1*
> * one input message and one output message: 541 ms
> * 8 input message and 30 output message per input message (240 output messages in total): 919 ms
>
> *Version 3.1.0*
> * one input message and one output message: 908 ms
> * 8 input message and 30 output message per input message (240 output messages in total): 6 sec 94 ms
>
> Even when the transformer just transforms and forwards one input message to one output message, the test takes approx. 400 ms longer to run.
> When transforming 8 input messages to 240 output messages it takes approx 5 seconds longer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)