[jira] [Reopened] (KAFKA-13183) Dropping nul key/value records upstream to repartiton topic not tracked via metrics

["Matthias J. Sax (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax reopened KAFKA-13183:
-------------------------------------

> Dropping nul key/value records upstream to repartiton topic not tracked via metrics
> -----------------------------------------------------------------------------------
>
>                 Key: KAFKA-13183
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13183
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Major
>
> For joins and aggregation, we consider records with null key or value as invalid, and drop them. Inside the aggregate and join processors, we record dropped record with a corresponding metric (cf `droppedRecrodSensor`).
> However, we also apply an upstream optimization if we need to repartition data. As we know that the downstream aggregation / join will drop those records anyway, we drop them _before_ we write them into the repartition topic (we still need the drop logic in the processor for the case we don't have a repartition topic).
> We add a `KStreamFilter` (cf `KStreamImpl#createRepartiitonSource()`) upstream but this filter does not update the corresponding metric to record dropped records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)