You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Javier Holguera <ja...@gmail.com> on 2019/11/12 17:40:15 UTC
Leveraging DLQ for errors coming from a sink connector plugin
Hi,
Looking at the Kafka Connect code, it seems that the built-in support for
DLQ queues only works for errors related to transformations and converters
(headers, key, and value).
I wonder if it has been considered (and maybe discarded) to use the same
mechanism for the call to the connector-plugin.put() operation. That way,
it would be possible for connector-plugins to leverage the same DLQ
semantics that Connect already implements without "reinventing the wheel"
themselves.
We have found ourselves in that situation with one in-house connector
plugin that we are building (HTTP Connector with extra specific bits). When
the connector plugin has tried X times to do its HTTP call, we want to
"offload" the record into a DLQ queue.
Any chances that this will be implemented?
Thanks.
Regards,
Javier.
Re: Leveraging DLQ for errors coming from a sink connector plugin
Posted by Javier Holguera <ja...@gmail.com>.
Any chance that somebody can shed some light on this?
Thanks!
On Tue, 12 Nov 2019 at 17:40, Javier Holguera <ja...@gmail.com>
wrote:
> Hi,
>
> Looking at the Kafka Connect code, it seems that the built-in support for
> DLQ queues only works for errors related to transformations and converters
> (headers, key, and value).
>
> I wonder if it has been considered (and maybe discarded) to use the same
> mechanism for the call to the connector-plugin.put() operation. That way,
> it would be possible for connector-plugins to leverage the same DLQ
> semantics that Connect already implements without "reinventing the wheel"
> themselves.
>
> We have found ourselves in that situation with one in-house connector
> plugin that we are building (HTTP Connector with extra specific bits). When
> the connector plugin has tried X times to do its HTTP call, we want to
> "offload" the record into a DLQ queue.
>
> Any chances that this will be implemented?
>
> Thanks.
>
> Regards,
> Javier.
>