[jira] [Commented] (KAFKA-21380) Failed state-transition rollback bypasses DLQ when delivery count limit is reached

["Sushant Mahajan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133770#comment-18133770 ] 

Sushant Mahajan commented on KAFKA-21380:
-----------------------------------------

This was deliberately avoided as a persister fail => a 5 time call to persister with exponential backoff failed. Now with DLQ again, the first step is ARCHIVING which requires a persister write. This too has good chance of failing (same partition key, so same broker). Hence the decision was taken to not stall the in-flight records queue over this.

> Failed state-transition rollback bypasses DLQ when delivery count limit is reached
> ----------------------------------------------------------------------------------
>
>                 Key: KAFKA-21380
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21380
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 4.5.0
>            Reporter: da Yi
>            Assignee: da Yi
>            Priority: Major
>
> When the DLQ flow is enabled for a share group (KIP-1191), records whose delivery
> count reaches the limit are moved to ARCHIVING on the normal release path so they
> are written to the DLQ topic (InFlightState.tryUpdateState with dlqSupportEnabled=true).
> However, the rollback path in InFlightState.completeStateTransition(false) - taken
> when a two-phase state transition fails (e.g. the persister write fails) and the
> acquisition lock has expired - archives the records directly:
>     state = previousState.deliveryCount() >= rollbackState.maxDeliveryCount ?
>         RecordState.ARCHIVED : RecordState.AVAILABLE;
> This bypasses the DLQ flow entirely: records that should have been dead-lettered are
> silently archived and never appear in the DLQ topic. Two related inconsistencies:
> 1. ARCHIVING (normal path) vs ARCHIVED (rollback path) for the same limit-reached
>    condition.
> 2. The normal path evaluates the limit with the delivery count including the current
>    delivery, while the rollback path uses previousState.deliveryCount() - the counts
>    differ by one.
> Note that completeStateTransition(boolean) has no access to the dlqSupportEnabled
> flag, so the rollback path cannot currently reach ARCHIVING.
> Reproduction: new unit test InFlightStateTest
> (testFailedStateTransitionWithExpiredLockDeadLettersWhenDlqSupported) fails on
> trunk and passes with the fix.
> Proposed fix: thread the DLQ-support flag through the rollback path, mirroring the
> normal transition path, and start the DLQ flow for batches that move to ARCHIVING
> in both the local rollback and the persister-failure paths.
> Two test scenarios (step 5 in the reproduction) verify:
> - normal release path at the limit moves records to ARCHIVING (baseline)
> - rollback with expired lock at the limit moves records to ARCHIVING when DLQ is
>   enabled, and to ARCHIVED when disabled



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)