You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pirow Engelbrecht <pi...@etion.co.za> on 2020/08/20 12:53:24 UTC

Kafka streams sink outputs weird records

Hello,

I've got Kafka Streams up and running with the following topology:
Sub-topology: 0
    Source: TopicInput (topics: [inputTopic])
      --> InputProcessor
    Processor: InputProcessor (stores: [KvStore])
      --> TopicOutput
      <-- TopicInput
    Source: KvInput (topics: [kvStoreTopic])
      --> KvProcessor
    Processor: KvProcessor (stores: [KvStore])
      --> none
      <-- KvInput
    Sink: TopicOutput (topic: outputTopic)
      <-- InputProcessor

There is only one context().forward(key,value) call from the InputProcessor to the TopicOutput sink. For some reason I get one additional Kafka record for each record the TopicOutput sinks puts into the output topic. They look like this:
ConsumerRecord(topic='outputTopic', partition=0, offset=1, timestamp=1597926832492, timestamp_type=0, key=b'\x00\x00\x00\x01', value=b'\x00\x00\x00\x00\x00\x00', headers=[], checksum=None, serialized_key_size=4, serialized_value_size=6, serialized_header_size=-1)

With a binary key and value (key and value seems to always be the same). Any ideas?

Thanks

Pirow Engelbrecht
System Engineer

E. pirow.engelbrecht@etion.co.za<fi...@etion.co.za>
T. +27 12 678 9740 (ext. 9879)
M. +27 63 148 3376

76 Regency Drive | Irene | Centurion | 0157<https://goo.gl/maps/v9ZbwjqpPyL2>
www.etion.co.za<https://www.parsec.co.za/>


[cid:image001.jpg@01D67701.73EE0660]<https://www.parsec.co.za/>


Facebook<https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr> | YouTube<https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A> | LinkedIn<https://www.linkedin.com/company/etionltd> | Twitter<https://twitter.com/Etionlimited> | Instagram<https://www.instagram.com/Etionlimited/>

Re: Kafka streams sink outputs weird records

Posted by Bruno Cadonna <br...@confluent.io>.
Hi Pirow,

hard to to have an idea without seeing the code that is executed in the 
processors.

Could you please post a minimal example that reproduces the issue?

Best,
Bruno

On 20.08.20 14:53, Pirow Engelbrecht wrote:
> Hello,
> 
> I’ve got Kafka Streams up and running with the following topology:
> 
> Sub-topology: 0
> 
>      Source: TopicInput (topics: [inputTopic])
> 
>        --> InputProcessor
> 
>      Processor: InputProcessor (stores: [KvStore])
> 
>        --> TopicOutput
> 
>        <-- TopicInput
> 
>      Source: KvInput (topics: [kvStoreTopic])
> 
>        --> KvProcessor
> 
>      Processor: KvProcessor (stores: [KvStore])
> 
>        --> none
> 
>        <-- KvInput
> 
>      Sink: TopicOutput (topic: outputTopic)
> 
>        <-- InputProcessor
> 
> There is only one context().forward(key,value) call from the 
> InputProcessor to the TopicOutput sink. For some reason I get one 
> additional Kafka record for each record the TopicOutput sinks puts into 
> the output topic. They look like this:
> 
> ConsumerRecord(topic='outputTopic', partition=0, offset=1, 
> timestamp=1597926832492, timestamp_type=0, key=b'\x00\x00\x00\x01', 
> value=b'\x00\x00\x00\x00\x00\x00', headers=[], checksum=None, 
> serialized_key_size=4, serialized_value_size=6, serialized_header_size=-1)
> 
> With a binary key and value (key and value seems to always be the same). 
> Any ideas?
> 
> Thanks
> 
> *Pirow Engelbrecht*
> System Engineer
> 
> *E.*pirow.engelbrecht@etion.co.za 
> <fi...@etion.co.za>
> *T.* +27 12 678 9740 (ext. 9879)
> *M.*+27 63 148 3376
> 
> 76 Regency Drive | Irene | Centurion | 0157 
> <https://goo.gl/maps/v9ZbwjqpPyL2>
> *www.etion.co.za <https://www.parsec.co.za/>*
> 
> <https://www.parsec.co.za/>
> 
> Facebook 
> <https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr> | 
> YouTube <https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A> | 
> LinkedIn <https://www.linkedin.com/company/etionltd> | Twitter 
> <https://twitter.com/Etionlimited> | Instagram 
> <https://www.instagram.com/Etionlimited/>
>