[jira] [Created] (KAFKA-8917) When performing a Left/Right-Join, pick the headers of the same side

["Pascal Büttiker (Jira)" <ji...@apache.org>]

Pascal Büttiker created KAFKA-8917:
--------------------------------------

             Summary: When performing a Left/Right-Join, pick the headers of the same side
                 Key: KAFKA-8917
                 URL: https://issues.apache.org/jira/browse/KAFKA-8917
             Project: Kafka
          Issue Type: Improvement
          Components: streams
            Reporter: Pascal Büttiker


As described in KAFKA-7718, headers are promoted by the "triggering" record in stateful operations such as Joins. This was very confusing and we spent quite some time debugging this.

While we ideally have full control over this behaviour as like the KAFKA-7718 proposes, I hope we can solve some of the randomness before this:
 * Inner-Join: Keep as is (use the headers of the triggering record)
 * Full-Join: Keep as is (use the headers of the triggering record)
 * Left-Join: *Always pick the headers of the left record.*
 * Right-Join: *Always pick the headers of the right record.*

This behaviour would solve the most pressing issues when dealing with headers in Kafka Streams.

*Motivation*:

We depend on headers, especially when dealing with tombstone records. There is no other way to store additional information. If we do not use tombstone records, all default Kafka Features around compacted topics and KTabels are no longer useable. We are able to use custom Transformers to generate the headers (basically patch in the missing header support in Kafka Streams), but as soon that we use Join/Aggregate we loose control over the headers.

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)