You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Akash Dhiman <ak...@gmail.com> on 2024/06/12 21:15:28 UTC

Making connector fault tolerant

Hello,
we have a requirement to make kafka connector more fault tolerant for our
use, where in we don't want them to fail for some kinds of errors, i.e
error where bootstrap broker are missing or if we don't have sufficient
permission to read data from topic (we are reading from aws msk).

we tried using basic error handing for such exception where in, in the poll
method we tried to swallow SaslException and config exceptions where
bootstrap brokers are missing, and retry after t amount of time but this
seems to make the connector make lots of unassigned task corresponding to
even a connector even with max.tasks set to 1.

1. unclear as to why that happens
2. looking for guidance on more standardised way to ensure resilient
connectors that don't transition to fail state on some errors which we
expect can happen

Re: Making connector fault tolerant

Posted by Akash Dhiman <ak...@gmail.com>.
yes, we create connectors on the fly and in some cases after a new
deployment msk cluster's public access is disabled thereby making the
bootstrap brokers present there unavailable to currently running connectors.

I am curious to know if it's possible to momentarily catch exceptions in
such cases without stopping or failing the connector tasks.

perhaps this is not the right solution but catching exceptions in the poll
method of connector have created strange behavior where it keeps on
spawning the tasks

On Thu, 13 Jun 2024, 03:40 Alex Craig, <al...@gmail.com> wrote:

> Hi Akash, if your connector doesn't have the appropriate permissions for a
> topic then it can't run - so I'm not sure what value there is in trying to
> handle or tolerate that kind of exception.  If someone is changing ACLs in
> such a way that it breaks a connector, then you probably want that
> connector to go into a failed state.  I don't think there's a way to change
> your connector code to handle this since consuming/producing to topics is
> handled by the Connect framework, but in my opinion if people are
> repeatedly breaking ACLs, then that might be better fixed with a process
> change?  (if this is an ongoing problem then it sounds like there's a
> larger issue happening that should be addressed)
>
> When you say "bootstrap brokers are missing"... are you referring to
> someone misconfiguring a connector such that the bootstrap servers config
> is missing?
>
> - alex
>
> On Wed, Jun 12, 2024 at 4:16 PM Akash Dhiman <ak...@gmail.com>
> wrote:
>
> > Hello,
> > we have a requirement to make kafka connector more fault tolerant for our
> > use, where in we don't want them to fail for some kinds of errors, i.e
> > error where bootstrap broker are missing or if we don't have sufficient
> > permission to read data from topic (we are reading from aws msk).
> >
> > we tried using basic error handing for such exception where in, in the
> poll
> > method we tried to swallow SaslException and config exceptions where
> > bootstrap brokers are missing, and retry after t amount of time but this
> > seems to make the connector make lots of unassigned task corresponding to
> > even a connector even with max.tasks set to 1.
> >
> > 1. unclear as to why that happens
> > 2. looking for guidance on more standardised way to ensure resilient
> > connectors that don't transition to fail state on some errors which we
> > expect can happen
> >
>

Re: Making connector fault tolerant

Posted by Alex Craig <al...@gmail.com>.
Hi Akash, if your connector doesn't have the appropriate permissions for a
topic then it can't run - so I'm not sure what value there is in trying to
handle or tolerate that kind of exception.  If someone is changing ACLs in
such a way that it breaks a connector, then you probably want that
connector to go into a failed state.  I don't think there's a way to change
your connector code to handle this since consuming/producing to topics is
handled by the Connect framework, but in my opinion if people are
repeatedly breaking ACLs, then that might be better fixed with a process
change?  (if this is an ongoing problem then it sounds like there's a
larger issue happening that should be addressed)

When you say "bootstrap brokers are missing"... are you referring to
someone misconfiguring a connector such that the bootstrap servers config
is missing?

- alex

On Wed, Jun 12, 2024 at 4:16 PM Akash Dhiman <ak...@gmail.com>
wrote:

> Hello,
> we have a requirement to make kafka connector more fault tolerant for our
> use, where in we don't want them to fail for some kinds of errors, i.e
> error where bootstrap broker are missing or if we don't have sufficient
> permission to read data from topic (we are reading from aws msk).
>
> we tried using basic error handing for such exception where in, in the poll
> method we tried to swallow SaslException and config exceptions where
> bootstrap brokers are missing, and retry after t amount of time but this
> seems to make the connector make lots of unassigned task corresponding to
> even a connector even with max.tasks set to 1.
>
> 1. unclear as to why that happens
> 2. looking for guidance on more standardised way to ensure resilient
> connectors that don't transition to fail state on some errors which we
> expect can happen
>