Re: [DISCUSS] KAFKA-4911 Add ProducerRecord to ProducerInterceptor#onAcknowledgement

[Kevin Conaway <ke...@gmail.com>]

Does anyone have any thoughts on this?  Should I create a proper KIP to
better facilitate discussion?

On Tue, Mar 21, 2017 at 11:05 AM, Kevin Conaway <ke...@gmail.com>
wrote:

> Greetings all,
>
> Please correct me if this is not the right place to discuss this.  I filed
> a jira about this but I figured the mailing list might encourage better
> collaboration.
>
> I'd like to propose adding the ProducerRecord field to the
> onAcknowledgement callback in the ProducerInterceptor interface that was
> added in KAFKA-3162
>
> The issue I'm having is that its not possible to determine if a specific
> record was sent (based on the record content or key), only that some record
> was sent based on the RecordMetadata.  From reading the KIP and the initial
> mailing list discussion, it seems that the design was modeled after the
> existing Producer.Callback interface. This model makes sense given that
> callbacks are typically anonymous functions where you would have access to
> the producer record:
>
>     ProducerRecord record = ...;
>     producer.send(record, new Callback() {
>       @Override
>       public void onCompletion(RecordMetadata recordMetadata, Exception e)
> {
>         // do something with record
>       }
>     })
>
> However, based on my understanding of the interceptor design, the
> interceptor should be thread safe and thus stateless. Even if you wanted to
> make it stateful, its not easy to tie together a record from onSend to
> onAcknowledgement.  Exception handling also suffers from this if you want
> to tie the exception to the specific record that failed
>
> With that in mind, is there a way that the producer interceptor API can be
> modified so that implementers can be aware of the key & content of the
> record that was acknowledged or failed?
>
> Thanks,
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>



-- 
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway