You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yoda Jedi Master <yo...@gmail.com> on 2021/08/10 08:25:00 UTC
Kafka Streams Handling uncaught exceptions REPLACE_THREAD
"REPLACE_THREAD - Replaces the thread receiving the exception and
processing continues with the same number of configured threads. (Note:
this can result in duplicate records depending on the application’s
processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
If an application gets an uncaught exception, then the failed thread will
be replaced with another thread and it will continue processing messages,
skipping that failed message?
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Posted by Yoda Jedi Master <yo...@gmail.com>.
Hi Luke, thank you for your answer.
I will try it, I think I will set an alert if there are too many messages.
To ignore the message should I simply return "replace_thread" in the
handler?
On Tue, Aug 10, 2021 at 12:16 PM Luke Chen <sh...@gmail.com> wrote:
> Hi Yoda,
> For your question:
> > If an application gets an uncaught exception, then the failed thread will
> be replaced with another thread and it will continue processing messages,
> skipping that failed message?
>
> --> Yes, if everything goes well after `replace thread`, you can ignore
> this failed message. Just one reminder that you should check the failed
> message to avoid this `uncaught exception` thrown again, because if this
> happens frequently, it'll impact application performance.
>
> Thank you.
> Luke
>
> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yo...@gmail.com>
> wrote:
>
> > "REPLACE_THREAD - Replaces the thread receiving the exception and
> > processing continues with the same number of configured threads. (Note:
> > this can result in duplicate records depending on the application’s
> > processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
> >
> > If an application gets an uncaught exception, then the failed thread will
> > be replaced with another thread and it will continue processing messages,
> > skipping that failed message?
> >
>
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Posted by Yoda Jedi Master <yo...@gmail.com>.
Thank you for your help, I will check it and try it :-)
On Mon, Aug 16, 2021 at 11:45 AM Bruno Cadonna <ca...@apache.org> wrote:
> Hi Yoda,
>
> for certain cases, Kafka Streams allows you to specify handlers that
> skip the problematic record. Those handlers are:
>
> 1. deserialization exception handler configured in
> default.deserialization.exception.handler
> 2. time extractor set in default.timestamp.extractor and in the Consumed
> object
> 3. production exception handler configured in
> default.production.exception.handler
>
> Kafka Streams provides implementations for handlers 1 and 2 to skip the
> problematic records, that are LogAndContinueExceptionHandler and
> LogAndSkipOnInvalidTimestamp, respectively.
>
> For some more details have a look at
>
> https://docs.confluent.io/platform/current/streams/faq.html#failure-and-exception-handling
>
> If problematic records cause an exception in user code, the user code
> needs to provide functionality to skip the problematic record.
>
> Best,
> Bruno
>
> On 10.08.21 13:26, Yoda Jedi Master wrote:
> > Hi Bruno, thank you for your answer.
> > I mean that the message that caused the exception was consumed and
> replaced
> > thread will continue from the next message. How then does it handle
> > uncaught exceptions, if it will fail again?
> >
> >
> > On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna <ca...@apache.org>
> wrote:
> >
> >> Hi Yoda,
> >>
> >> What do you mean exactly with "skipping that failed message"?
> >>
> >> Do you mean a record consumed from a topic that caused an exception that
> >> killed the stream thread?
> >>
> >> If the record killed the stream thread due to an exception, for example,
> >> a deserialization exception, it will probably also kill the next stream
> >> thread that will read that record. Replacing a stream thread does not
> >> skip records but it can result in duplicate records depending on the
> >> application’s processing mode determined by the
> >> PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited.
> >>
> >> Best,
> >> Bruno
> >>
> >>
> >>
> >> On 10.08.21 11:15, Luke Chen wrote:
> >>> Hi Yoda,
> >>> For your question:
> >>>> If an application gets an uncaught exception, then the failed thread
> >> will
> >>> be replaced with another thread and it will continue processing
> messages,
> >>> skipping that failed message?
> >>>
> >>> --> Yes, if everything goes well after `replace thread`, you can ignore
> >>> this failed message. Just one reminder that you should check the failed
> >>> message to avoid this `uncaught exception` thrown again, because if
> this
> >>> happens frequently, it'll impact application performance.
> >>>
> >>> Thank you.
> >>> Luke
> >>>
> >>> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yo...@gmail.com>
> >> wrote:
> >>>
> >>>> "REPLACE_THREAD - Replaces the thread receiving the exception and
> >>>> processing continues with the same number of configured threads.
> (Note:
> >>>> this can result in duplicate records depending on the application’s
> >>>> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
> >>>>
> >>>> If an application gets an uncaught exception, then the failed thread
> >> will
> >>>> be replaced with another thread and it will continue processing
> >> messages,
> >>>> skipping that failed message?
> >>>>
> >>>
> >>
> >
>
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Posted by Bruno Cadonna <ca...@apache.org>.
Hi Yoda,
for certain cases, Kafka Streams allows you to specify handlers that
skip the problematic record. Those handlers are:
1. deserialization exception handler configured in
default.deserialization.exception.handler
2. time extractor set in default.timestamp.extractor and in the Consumed
object
3. production exception handler configured in
default.production.exception.handler
Kafka Streams provides implementations for handlers 1 and 2 to skip the
problematic records, that are LogAndContinueExceptionHandler and
LogAndSkipOnInvalidTimestamp, respectively.
For some more details have a look at
https://docs.confluent.io/platform/current/streams/faq.html#failure-and-exception-handling
If problematic records cause an exception in user code, the user code
needs to provide functionality to skip the problematic record.
Best,
Bruno
On 10.08.21 13:26, Yoda Jedi Master wrote:
> Hi Bruno, thank you for your answer.
> I mean that the message that caused the exception was consumed and replaced
> thread will continue from the next message. How then does it handle
> uncaught exceptions, if it will fail again?
>
>
> On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna <ca...@apache.org> wrote:
>
>> Hi Yoda,
>>
>> What do you mean exactly with "skipping that failed message"?
>>
>> Do you mean a record consumed from a topic that caused an exception that
>> killed the stream thread?
>>
>> If the record killed the stream thread due to an exception, for example,
>> a deserialization exception, it will probably also kill the next stream
>> thread that will read that record. Replacing a stream thread does not
>> skip records but it can result in duplicate records depending on the
>> application’s processing mode determined by the
>> PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited.
>>
>> Best,
>> Bruno
>>
>>
>>
>> On 10.08.21 11:15, Luke Chen wrote:
>>> Hi Yoda,
>>> For your question:
>>>> If an application gets an uncaught exception, then the failed thread
>> will
>>> be replaced with another thread and it will continue processing messages,
>>> skipping that failed message?
>>>
>>> --> Yes, if everything goes well after `replace thread`, you can ignore
>>> this failed message. Just one reminder that you should check the failed
>>> message to avoid this `uncaught exception` thrown again, because if this
>>> happens frequently, it'll impact application performance.
>>>
>>> Thank you.
>>> Luke
>>>
>>> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yo...@gmail.com>
>> wrote:
>>>
>>>> "REPLACE_THREAD - Replaces the thread receiving the exception and
>>>> processing continues with the same number of configured threads. (Note:
>>>> this can result in duplicate records depending on the application’s
>>>> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
>>>>
>>>> If an application gets an uncaught exception, then the failed thread
>> will
>>>> be replaced with another thread and it will continue processing
>> messages,
>>>> skipping that failed message?
>>>>
>>>
>>
>
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Posted by Yoda Jedi Master <yo...@gmail.com>.
Hi Bruno, thank you for your answer.
I mean that the message that caused the exception was consumed and replaced
thread will continue from the next message. How then does it handle
uncaught exceptions, if it will fail again?
On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna <ca...@apache.org> wrote:
> Hi Yoda,
>
> What do you mean exactly with "skipping that failed message"?
>
> Do you mean a record consumed from a topic that caused an exception that
> killed the stream thread?
>
> If the record killed the stream thread due to an exception, for example,
> a deserialization exception, it will probably also kill the next stream
> thread that will read that record. Replacing a stream thread does not
> skip records but it can result in duplicate records depending on the
> application’s processing mode determined by the
> PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited.
>
> Best,
> Bruno
>
>
>
> On 10.08.21 11:15, Luke Chen wrote:
> > Hi Yoda,
> > For your question:
> >> If an application gets an uncaught exception, then the failed thread
> will
> > be replaced with another thread and it will continue processing messages,
> > skipping that failed message?
> >
> > --> Yes, if everything goes well after `replace thread`, you can ignore
> > this failed message. Just one reminder that you should check the failed
> > message to avoid this `uncaught exception` thrown again, because if this
> > happens frequently, it'll impact application performance.
> >
> > Thank you.
> > Luke
> >
> > On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yo...@gmail.com>
> wrote:
> >
> >> "REPLACE_THREAD - Replaces the thread receiving the exception and
> >> processing continues with the same number of configured threads. (Note:
> >> this can result in duplicate records depending on the application’s
> >> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
> >>
> >> If an application gets an uncaught exception, then the failed thread
> will
> >> be replaced with another thread and it will continue processing
> messages,
> >> skipping that failed message?
> >>
> >
>
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Posted by Bruno Cadonna <ca...@apache.org>.
Hi Yoda,
What do you mean exactly with "skipping that failed message"?
Do you mean a record consumed from a topic that caused an exception that
killed the stream thread?
If the record killed the stream thread due to an exception, for example,
a deserialization exception, it will probably also kill the next stream
thread that will read that record. Replacing a stream thread does not
skip records but it can result in duplicate records depending on the
application’s processing mode determined by the
PROCESSING_GUARANTEE_CONFIG value as stated in the docs you cited.
Best,
Bruno
On 10.08.21 11:15, Luke Chen wrote:
> Hi Yoda,
> For your question:
>> If an application gets an uncaught exception, then the failed thread will
> be replaced with another thread and it will continue processing messages,
> skipping that failed message?
>
> --> Yes, if everything goes well after `replace thread`, you can ignore
> this failed message. Just one reminder that you should check the failed
> message to avoid this `uncaught exception` thrown again, because if this
> happens frequently, it'll impact application performance.
>
> Thank you.
> Luke
>
> On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yo...@gmail.com> wrote:
>
>> "REPLACE_THREAD - Replaces the thread receiving the exception and
>> processing continues with the same number of configured threads. (Note:
>> this can result in duplicate records depending on the application’s
>> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
>>
>> If an application gets an uncaught exception, then the failed thread will
>> be replaced with another thread and it will continue processing messages,
>> skipping that failed message?
>>
>
Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD
Posted by Luke Chen <sh...@gmail.com>.
Hi Yoda,
For your question:
> If an application gets an uncaught exception, then the failed thread will
be replaced with another thread and it will continue processing messages,
skipping that failed message?
--> Yes, if everything goes well after `replace thread`, you can ignore
this failed message. Just one reminder that you should check the failed
message to avoid this `uncaught exception` thrown again, because if this
happens frequently, it'll impact application performance.
Thank you.
Luke
On Tue, Aug 10, 2021 at 4:25 PM Yoda Jedi Master <yo...@gmail.com> wrote:
> "REPLACE_THREAD - Replaces the thread receiving the exception and
> processing continues with the same number of configured threads. (Note:
> this can result in duplicate records depending on the application’s
> processing mode determined by the PROCESSING_GUARANTEE_CONFIG value)"
>
> If an application gets an uncaught exception, then the failed thread will
> be replaced with another thread and it will continue processing messages,
> skipping that failed message?
>