You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Victor Osorio <Vi...@amdocs.com.INVALID> on 2025/09/03 17:50:56 UTC
Graceful Shutdown on Kafka Streams exception handler
Hello everyone,
We're currently using Kafka Streams to process transactional data with exactly-once semantics (EOS). However, for some of our workloads, we require higher throughput, which makes EOS impractical.
To ensure data integrity, we rely on UncaughtExceptionHandler and ProductionExceptionHandler to halt stream processing upon any exception. This prevents data loss but introduces a new challenge: when a thread stops due to an exception, it doesn't commit the records that were already successfully processed. As a result, when the stream restarts, those records are reprocessed, leading to duplication.
While reviewing the discussion around KIP-1033, I noticed the suggestion to avoid exposing commit functionality in the Kafka Streams API (https://lists.apache.org/thread/k4v0737tqjdnq5vl3yp9rjr4qzqoo306). That makes sense in many contexts, but I'd like to revisit a related idea:
Could we introduce a new shutdown mechanism, perhaps a "Graceful Shutdown" API, that commits all successfully processed records while skipping the one that caused the failure?
This would allow us to maintain data integrity without sacrificing throughput or introducing duplicates. I'm curious to hear your thoughts:
* Would this be possible to implement with current Kafka Streams APIs?
* Is that possible, or desired, to be added as a Kafka Streams feature in further releases? If yes, we can open a KIP.
Looking forward to your insights and feedback.
Best regards,
Victor Osório
[amdocs-2017-brand-mark-rgb]
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
[DISCUSS] KIP-1215 Graceful stop on Kafka Streams exceptions processing handlers
Posted by Victor Osorio <Vi...@amdocs.com.INVALID>.
Thanks Matthias,
I would like to start a discussion thread on KIP-1215.
I'm proposing add a new option to ProcessingExceptionHandler where the application before fail, commit all previous records to avoid duplicated records if streams process restart automatically.
Please take a look and feel free to share any thoughts.
https://cwiki.apache.org/confluence/display/KAFKA/KIP+1215%3A+Graceful+stop+on+Kafka+Streams+exceptions+processing+handlers
Thanks,
Victor Osório
-----Original Message-----
From: Matthias J. Sax <mj...@apache.org>
Sent: Wednesday, September 3, 2025 10:00 PM
To: users@kafka.apache.org
Subject: Re: Graceful Shutdown on Kafka Streams exception handler
[You don't often get email from mjsax@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
CAUTION: This email is from an external source. Please don't open any unknown links or attachments.
Hello Victor,
thanks for reaching out. I am not sure if it would be easily possible to implement what you propose. The problem is really, that records might be partially processed when an error happens, and before we can figure out what the correct offset for committing is, KS needs to flush, but after we hit an error, flushing cleanly might not be possible any longer.
Beside the above issue, w/o EOS (ie, Kafka transactions), there is all kind of other scenarios that could lead to duplicate output, or reprocessing an input record a second time. So even if we can implement what you propose (we would need to investigate in more details if it might be possible or not), it could only reduce duplicates, but not fully eliminate them. Just want to make sure you are aware of this.
So if you are really interested to contribute such a feature, and you can figure out how to do this correctly, yes, please write a KIP about it :) -- We can also try to help you with this investigation (it might require some POC PR...); however, not sure how much time we can find atm to support you TBH.
However, I am wondering what kind of performance you need, and why EOS would not be able to deliver. There is many configs, so maybe there is a way to tune your app to make it work with EOS?
-Matthias
On 9/3/25 10:50 AM, Victor Osorio wrote:
> Hello everyone,
>
> We're currently using Kafka Streams to process transactional data with
> *exactly-once semantics (EOS)*. However, for some of our workloads, we
> require higher throughput, which makes EOS impractical.
>
> To ensure data integrity, we rely
> on UncaughtExceptionHandler and ProductionExceptionHandler to halt
> stream processing upon any exception. This prevents data loss but
> introduces a new challenge: when a thread stops due to an exception,
> it doesn't commit the records that were already successfully
> processed. As a result, when the stream restarts, those records are
> reprocessed, leading to duplication.
>
> While reviewing the discussion around KIP-1033, I noticed the
> suggestion to avoid exposing commit functionality in the Kafka Streams
> API
> (https://lis/
> ts.apache.org%2Fthread%2Fk4v0737tqjdnq5vl3yp9rjr4qzqoo306&data=05%7C02
> %7CVictor.Osorio%40amdocs.com%7C49feb9f09642470bfed308ddeb4e6349%7Cc8e
> ca3ca127646d59d9da0f2a028920f%7C0%7C0%7C638925444117688366%7CUnknown%7
> CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zM
> iIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=uQGZx2cHVjWZ%2F
> KZaX4z2sXSprz5d6%2BuoN83XntX%2FT9g%3D&reserved=0
> <https://lists.apache.org/thread/k4v0737tqjdnq5vl3yp9rjr4qzqoo306>).
> That makes sense in many contexts, but I'd like to revisit a related idea:
>
> *Could we introduce a new shutdown mechanism, perhaps a "Graceful
> Shutdown" API, that commits all successfully processed records while
> skipping the one that caused the failure?*
>
> This would allow us to maintain data integrity without sacrificing
> throughput or introducing duplicates. I'm curious to hear your thoughts:
>
> * Would this be possible to implement with current Kafka Streams APIs?
> * Is that possible, or desired, to be added as a Kafka Streams feature
> in further releases? If yes, we can open a KIP.
>
> Looking forward to your insights and feedback.
>
> Best regards,
> Victor Osório
>
> amdocs-2017-brand-mark-rgb
>
> *This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which
> you may review
> at**https://www.amdocs.com/about/email-terms-of-service*
> <https://www.amdocs.com/about/email-terms-of-service>
>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
Re: Graceful Shutdown on Kafka Streams exception handler
Posted by "Matthias J. Sax" <mj...@apache.org>.
Hello Victor,
thanks for reaching out. I am not sure if it would be easily possible to
implement what you propose. The problem is really, that records might be
partially processed when an error happens, and before we can figure out
what the correct offset for committing is, KS needs to flush, but after
we hit an error, flushing cleanly might not be possible any longer.
Beside the above issue, w/o EOS (ie, Kafka transactions), there is all
kind of other scenarios that could lead to duplicate output, or
reprocessing an input record a second time. So even if we can implement
what you propose (we would need to investigate in more details if it
might be possible or not), it could only reduce duplicates, but not
fully eliminate them. Just want to make sure you are aware of this.
So if you are really interested to contribute such a feature, and you
can figure out how to do this correctly, yes, please write a KIP about
it :) -- We can also try to help you with this investigation (it might
require some POC PR...); however, not sure how much time we can find atm
to support you TBH.
However, I am wondering what kind of performance you need, and why EOS
would not be able to deliver. There is many configs, so maybe there is a
way to tune your app to make it work with EOS?
-Matthias
On 9/3/25 10:50 AM, Victor Osorio wrote:
> Hello everyone,
>
> We’re currently using Kafka Streams to process transactional data with
> *exactly-once semantics (EOS)*. However, for some of our workloads, we
> require higher throughput, which makes EOS impractical.
>
> To ensure data integrity, we rely
> on UncaughtExceptionHandler and ProductionExceptionHandler to halt
> stream processing upon any exception. This prevents data loss but
> introduces a new challenge: when a thread stops due to an exception, it
> doesn’t commit the records that were already successfully processed. As
> a result, when the stream restarts, those records are reprocessed,
> leading to duplication.
>
> While reviewing the discussion around KIP-1033, I noticed the suggestion
> to avoid exposing commit functionality in the Kafka Streams API
> (https://lists.apache.org/thread/k4v0737tqjdnq5vl3yp9rjr4qzqoo306
> <https://lists.apache.org/thread/k4v0737tqjdnq5vl3yp9rjr4qzqoo306>).
> That makes sense in many contexts, but I’d like to revisit a related idea:
>
> *Could we introduce a new shutdown mechanism, perhaps a “Graceful
> Shutdown” API, that commits all successfully processed records while
> skipping the one that caused the failure?*
>
> This would allow us to maintain data integrity without sacrificing
> throughput or introducing duplicates. I’m curious to hear your thoughts:
>
> * Would this be possible to implement with current Kafka Streams APIs?
> * Is that possible, or desired, to be added as a Kafka Streams feature
> in further releases? If yes, we can open a KIP.
>
> Looking forward to your insights and feedback.
>
> Best regards,
> Victor Osório
>
> amdocs-2017-brand-mark-rgb
>
> *This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which you
> may review at**https://www.amdocs.com/about/email-terms-of-service*
> <https://www.amdocs.com/about/email-terms-of-service>
>