[jira] [Created] (KAFKA-13183) Dropping nul key/value records upstream to repartiton topic not tracked via metrics

["Matthias J. Sax (Jira)" <ji...@apache.org>]

Matthias J. Sax created KAFKA-13183:
---------------------------------------

             Summary: Dropping nul key/value records upstream to repartiton topic not tracked via metrics
                 Key: KAFKA-13183
                 URL: https://issues.apache.org/jira/browse/KAFKA-13183
             Project: Kafka
          Issue Type: Bug
          Components: streams
            Reporter: Matthias J. Sax


For joins and aggregation, we consider records with null key or value as invalid, and drop them. Inside the aggregate and join processors, we record dropped record with a corresponding metric (cf `droppedRecrodSensor`).

However, we also apply an upstream optimization if we need to repartition data. As we know that the downstream aggregation / join will drop those records anyway, we drop them _before_ we write them into the repartition topic (we still need the drop logic in the processor for the case we don't have a repartition topic).

We add a `KStreamFilter` (cf `KStreamImpl#createRepartiitonSource()`) upstream but this filter does not update the corresponding metric to record dropped records.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)