where to capture a failed task's exception

[Akash Dhiman <ak...@gmail.com>]

Hello,

I have a requirement where I need to detect failed tasks based on the
specific errors and emit a metric only when it doesn't fail based on these
specific errors (these include unknown_topic_or_partition, specific cases
of ConfigException etc)

I know about a similar metric accessible via Prometheus but that gives me
count of failed task count for any reason.

I was thinking that wrapping the poll method of the task in try catch block
would be sufficient where i detect for error details in the catch block and
emit metric when they don't match the ones i don't want the metric for)

but I am unsure if this captures all possible scenarios for which a task
might fail.

is it guaranteed that all the exceptions/error for which a task might fail
gets emitted via the poll method?

Re: where to capture a failed task's exception

[Greg Harris <gr...@aiven.io.INVALID>]

Hey Akash,

Thanks for the question! For a direct answer, no: throwing exceptions
from poll() is only one of many ways that a task can fail.

If you look at the AK source, every failure ultimately uses the
AbstractStatus.State.FAILED enum [1]. You can trace the usages of this
enum back to see all of the ways that a connector or task can fail.
The failure trace is also exposed in the REST API, which is a stable
public API you can depend on.

[1]: https://github.com/apache/kafka/blob/d582d5aff517879b150bc2739bad99df07e15e2b/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractStatus.java#L27C13-L27C13

Happy to help,
Greg

On Tue, Dec 26, 2023 at 2:41 PM Akash Dhiman <ak...@gmail.com> wrote:
>
> Hello,
>
> I have a requirement where I need to detect failed tasks based on the
> specific errors and emit a metric only when it doesn't fail based on these
> specific errors (these include unknown_topic_or_partition, specific cases
> of ConfigException etc)
>
> I know about a similar metric accessible via Prometheus but that gives me
> count of failed task count for any reason.
>
> I was thinking that wrapping the poll method of the task in try catch block
> would be sufficient where i detect for error details in the catch block and
> emit metric when they don't match the ones i don't want the metric for)
>
> but I am unsure if this captures all possible scenarios for which a task
> might fail.
>
> is it guaranteed that all the exceptions/error for which a task might fail
> gets emitted via the poll method?