Kafka Connect HdfsSink and Poison Messages

["Tauzell, Dave" <Da...@surescripts.com>]

I have the HdfsSink reading from a topic.  If it finds a message that it cannot deal with then it stops processing.  Is there a way to tell the HdfsSink to put that message somewhere else and continue on?  In other queuing systems this is referred to as  "Dead Letter Queue".

-Dave
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.

Re: Kafka Connect HdfsSink and Poison Messages

[Ewen Cheslack-Postava <ew...@confluent.io>]

Not today, although that's something we might want to add support for at
the framework level (publish to a Kafka dead letter topic) and just provide
hooks for to sinks so they don't all have to handle that case.

Today the solution would be to reconfigure your connector/worker so it can
handle the message. An alternative would be to manually force offsets to a
new position, effectively skipping the message. This is also something we'd
like to expose, although this isn't even easy for regular clients today and
Connect has to deal with both Kafka offset (for sinks) and Connect offsets
(for sources).

-Ewen

On Tue, Jun 14, 2016 at 11:56 AM, Tauzell, Dave <
Dave.Tauzell@surescripts.com> wrote:

> I have the HdfsSink reading from a topic.  If it finds a message that it
> cannot deal with then it stops processing.  Is there a way to tell the
> HdfsSink to put that message somewhere else and continue on?  In other
> queuing systems this is referred to as  "Dead Letter Queue".
>
> -Dave
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>



-- 
Thanks,
Ewen