You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Aditya Kousik <ad...@icloud.com> on 2026/04/01 15:13:55 UTC

[DISCUSS] KIP-1305: Per-Record Context Processing for Kafka Consumers

Hi all,

I'd like to start a discussion on KIP-1305: Per-Record Context Processing for Kafka Consumers.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1305%3A+Per-Record+Context+Processing+for+Kafka+Consumers

Spring Kafka, SmallRye, Micronaut, OTel, and Datadog have each independently built per-record lifecycle hooks because the client doesn't provide one. This KIP proposes ConsumerRecordContextProcessor, a configurable hook that creates a per-record AutoCloseable context via record.newContext(), giving frameworks and instrumentation libraries a single stable integration point.

Looking forward to your feedback.

Thanks,
Aditya Kousik

Re: [DISCUSS] KIP-1305: Per-Record Context Processing for Kafka Consumers

Posted by Aditya Kousik <ad...@icloud.com>.

Hi everyone,

Gentle nudge on this. Motivation is still a driving factor for this KIP. Spring, smallrye, micronaut (and to an extent ShareConsumers with an ack) all implement a per-record ack. 

It would be good to support a 1P approach at the client level so users can add hooks around the scope of consuming a single record (rather than just the batch as is with the current consumer interceptor). This helps cut boilerplate around instrumentation, mdc context setup etc.

Cheers,
Aditya

> On May 17, 2026, at 14:58, Aditya Kousik <ad...@icloud.com> wrote:
> 
> Hi all,
> 
> Adding on a couple more points for motivation:
> 
> 1. Users typically set MDC context from each record but this can break in subtle ways for both sync/async threading models
> 
> for (var r: consumer.poll(…)) {
>    MDC.put(“kafka.offset”, r.offset());
>    // add rest of the fields
>    process(r); // throws an ex
>    MDC.clear();
> }
> 
> In both sync and async models, record N+1 carries prev record’s context during exceptions. The solution is a try finally for cleanup, yet more boilerplate for teams to maintain manually. Across thread boundaries, we have to manually copy threadlocals around from the poller loop (such as OTel,DD spans).
> 
> In KafkaShareConsumer, this behaviour is more common when worker threads can perform individual redelivery, ack calls. Sometimes, a main poll loop for try/catch does not neatly cleanup state between records.
> 
> 2. I also see Confluent’s parallel-consumer implement a PollContext to achieve a similar function to wrap each poll() action per-record.
> 
> Best,
> Aditya
> 
>> On Apr 16, 2026, at 11:45, Aditya Kousik <ad...@icloud.com> wrote:
>> 
>> Hi kafka dev,
>> 
>> Gentle ping on this KIP. I’ve added a short [unit test reproducer](https://github.com/apache/kafka/pull/22077) that highlights the gap mentioned in the KIP.
>> 
>> Would appreciate any eyes on both.
>> 
>> Thanks,
>> Aditya
>> 
>>>> On Apr 1, 2026, at 08:14, Aditya Kousik <ad...@icloud.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> I'd like to start a discussion on KIP-1305: Per-Record Context Processing for Kafka Consumers.
>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1305%3A+Per-Record+Context+Processing+for+Kafka+Consumers
>>> 
>>> Spring Kafka, SmallRye, Micronaut, OTel, and Datadog have each independently built per-record lifecycle hooks because the client doesn't provide one. This KIP proposes ConsumerRecordContextProcessor, a configurable hook that creates a per-record AutoCloseable context via record.newContext(), giving frameworks and instrumentation libraries a single stable integration point.
>>> 
>>> Looking forward to your feedback.
>>> 
>>> Thanks,
>>> Aditya Kousik
>

Re: [DISCUSS] KIP-1305: Per-Record Context Processing for Kafka Consumers

Posted by Aditya Kousik <ad...@icloud.com>.

Hi all,

Adding on a couple more points for motivation:

1. Users typically set MDC context from each record but this can break in subtle ways for both sync/async threading models

for (var r: consumer.poll(…)) {
    MDC.put(“kafka.offset”, r.offset()); 
    // add rest of the fields
    process(r); // throws an ex
    MDC.clear();
}

In both sync and async models, record N+1 carries prev record’s context during exceptions. The solution is a try finally for cleanup, yet more boilerplate for teams to maintain manually. Across thread boundaries, we have to manually copy threadlocals around from the poller loop (such as OTel,DD spans).

In KafkaShareConsumer, this behaviour is more common when worker threads can perform individual redelivery, ack calls. Sometimes, a main poll loop for try/catch does not neatly cleanup state between records.

2. I also see Confluent’s parallel-consumer implement a PollContext to achieve a similar function to wrap each poll() action per-record.

Best,
Aditya

> On Apr 16, 2026, at 11:45, Aditya Kousik <ad...@icloud.com> wrote:
> 
> Hi kafka dev,
> 
> Gentle ping on this KIP. I’ve added a short [unit test reproducer](https://github.com/apache/kafka/pull/22077) that highlights the gap mentioned in the KIP. 
> 
> Would appreciate any eyes on both.
> 
> Thanks,
> Aditya
> 
>> On Apr 1, 2026, at 08:14, Aditya Kousik <ad...@icloud.com> wrote:
>> 
>> Hi all,
>> 
>> I'd like to start a discussion on KIP-1305: Per-Record Context Processing for Kafka Consumers.
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1305%3A+Per-Record+Context+Processing+for+Kafka+Consumers
>> 
>> Spring Kafka, SmallRye, Micronaut, OTel, and Datadog have each independently built per-record lifecycle hooks because the client doesn't provide one. This KIP proposes ConsumerRecordContextProcessor, a configurable hook that creates a per-record AutoCloseable context via record.newContext(), giving frameworks and instrumentation libraries a single stable integration point.
>> 
>> Looking forward to your feedback.
>> 
>> Thanks,
>> Aditya Kousik

Re: [DISCUSS] KIP-1305: Per-Record Context Processing for Kafka Consumers

Posted by Aditya Kousik <ad...@icloud.com>.

Hi kafka dev,

Gentle ping on this KIP. I’ve added a short [unit test reproducer](https://github.com/apache/kafka/pull/22077) that highlights the gap mentioned in the KIP. 

Would appreciate any eyes on both.

Thanks,
Aditya

> On Apr 1, 2026, at 08:14, Aditya Kousik <ad...@icloud.com> wrote:
> 
> Hi all,
> 
> I'd like to start a discussion on KIP-1305: Per-Record Context Processing for Kafka Consumers.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1305%3A+Per-Record+Context+Processing+for+Kafka+Consumers
> 
> Spring Kafka, SmallRye, Micronaut, OTel, and Datadog have each independently built per-record lifecycle hooks because the client doesn't provide one. This KIP proposes ConsumerRecordContextProcessor, a configurable hook that creates a per-record AutoCloseable context via record.newContext(), giving frameworks and instrumentation libraries a single stable integration point.
> 
> Looking forward to your feedback.
> 
> Thanks,
> Aditya Kousik