[jira] [Commented] (KAFKA-20113) Add Configurable Retry Parameters for Status Backing Store

["Mickael Maison (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18072284#comment-18072284 ] 

Mickael Maison commented on KAFKA-20113:
----------------------------------------

Moving to the next release as we're now in code freeze for 4.3.0.

> Add Configurable Retry Parameters for Status Backing Store
> ----------------------------------------------------------
>
>                 Key: KAFKA-20113
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20113
>             Project: Kafka
>          Issue Type: New Feature
>          Components: connect
>            Reporter: Said BOUDJELDA
>            Assignee: Said BOUDJELDA
>            Priority: Major
>              Labels: configuration, connect, improvement, reliability
>             Fix For: 4.3.0
>
>
> Implement configurable retry parameters for the +KafkaStatusBackingStore+ to address the TODO comment "retry more gracefully and not forever" and provide operators with control over retry behavior during transient failures.
> h3. Problem Statement
>  
> KafkaStatusBackingStore currently retries status updates indefinitely when encountering retriable exceptions. This behavior is problematic because:
>  # *Infinite retry loops* can cause the worker to become unresponsive during extended Kafka broker outages
>  # *No visibility* into retry behavior - operators cannot tune retry parameters based on their environment
>  # *Resource exhaustion* - indefinite retries can consume threads and memory during prolonged failures
>  # *No graceful degradation* - the system continues retrying without bound rather than failing fast when appropriate
> A TODO comment in the codebase ({{{}// TODO: retry more gracefully and not forever{}}}) explicitly acknowledges this issue needs addressing.
> h3. Proposed Solution
> Add four new configuration properties under the {{status.storage.}} prefix to control retry behavior:
>  
> ||Property||Type||Default||Description||
> |{{status.storage.retry.max.retries}}|INT|5|Maximum number of retry attempts before giving up|
> |{{status.storage.retry.initial.backoff.ms}}|LONG|300|Initial backoff delay in milliseconds|
> |{{status.storage.retry.max.backoff.ms}}|LONG|10000|Maximum backoff delay cap in milliseconds|
> |{{status.storage.retry.backoff.multiplier}}|DOUBLE|2.0|Multiplier applied to backoff after each attempt|
> The retry mechanism uses *exponential backoff with jitter* to prevent thundering herd problems during cluster recovery.
> h4. Behavior
>  * Retries occur only for exceptions marked as {{RetriableException}}
>  * After exhausting {{{}max.retries{}}}, the operation logs an error and terminates gracefully
>  * All retry attempts are logged at WARN level with attempt count and delay information
>  * Non-retriable exceptions fail immediately without retry
>  *  
> h3. Benefits
>  # *Predictable failure modes* - Workers eventually give up and surface errors instead of hanging
>  # *Operator control* - Tune retry behavior based on environment characteristics
>  # *Better observability* - Clear logging of retry attempts and outcomes
>  # *Backward compatible* - Default values maintain similar behavior to current implementation
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)