Re: [DISCUSS] KIP-1305: Per-Record Context Processing for Kafka Consumers

[Aditya Kousik <ad...@icloud.com>]

Hi everyone,

Gentle nudge on this. Motivation is still a driving factor for this KIP. Spring, smallrye, micronaut (and to an extent ShareConsumers with an ack) all implement a per-record ack. 

It would be good to support a 1P approach at the client level so users can add hooks around the scope of consuming a single record (rather than just the batch as is with the current consumer interceptor). This helps cut boilerplate around instrumentation, mdc context setup etc.

Cheers,
Aditya

> On May 17, 2026, at 14:58, Aditya Kousik <ad...@icloud.com> wrote:
> 
> Hi all,
> 
> Adding on a couple more points for motivation:
> 
> 1. Users typically set MDC context from each record but this can break in subtle ways for both sync/async threading models
> 
> for (var r: consumer.poll(…)) {
>    MDC.put(“kafka.offset”, r.offset());
>    // add rest of the fields
>    process(r); // throws an ex
>    MDC.clear();
> }
> 
> In both sync and async models, record N+1 carries prev record’s context during exceptions. The solution is a try finally for cleanup, yet more boilerplate for teams to maintain manually. Across thread boundaries, we have to manually copy threadlocals around from the poller loop (such as OTel,DD spans).
> 
> In KafkaShareConsumer, this behaviour is more common when worker threads can perform individual redelivery, ack calls. Sometimes, a main poll loop for try/catch does not neatly cleanup state between records.
> 
> 2. I also see Confluent’s parallel-consumer implement a PollContext to achieve a similar function to wrap each poll() action per-record.
> 
> Best,
> Aditya
> 
>> On Apr 16, 2026, at 11:45, Aditya Kousik <ad...@icloud.com> wrote:
>> 
>> Hi kafka dev,
>> 
>> Gentle ping on this KIP. I’ve added a short [unit test reproducer](https://github.com/apache/kafka/pull/22077) that highlights the gap mentioned in the KIP.
>> 
>> Would appreciate any eyes on both.
>> 
>> Thanks,
>> Aditya
>> 
>>>> On Apr 1, 2026, at 08:14, Aditya Kousik <ad...@icloud.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> I'd like to start a discussion on KIP-1305: Per-Record Context Processing for Kafka Consumers.
>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1305%3A+Per-Record+Context+Processing+for+Kafka+Consumers
>>> 
>>> Spring Kafka, SmallRye, Micronaut, OTel, and Datadog have each independently built per-record lifecycle hooks because the client doesn't provide one. This KIP proposes ConsumerRecordContextProcessor, a configurable hook that creates a per-record AutoCloseable context via record.newContext(), giving frameworks and instrumentation libraries a single stable integration point.
>>> 
>>> Looking forward to your feedback.
>>> 
>>> Thanks,
>>> Aditya Kousik
>