You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ritika Reddy (Jira)" <ji...@apache.org> on 2024/11/14 23:24:00 UTC
[jira] [Updated] (KAFKA-18019) Convert INVALID_PRODUCER_ID_MAPPING from abortable error to fatal error
[ https://issues.apache.org/jira/browse/KAFKA-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ritika Reddy updated KAFKA-18019:
---------------------------------
Description:
Since we bump epoch on abort, we no longer need to call InitProducerId to fence requests. InitProducerId will only be called when the producer starts up to fence a previous instance.
With this change, some other calls to InitProducerId were inspected including the call after receiving an InvalidPidMappingException. This exception was changed to abortable as part of [KIP-360: Improve reliability of idempotent/transactional producer|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820]. However, this change means that we can violate EOS guarantees. As an example:
Consider an application that is copying data from one partition to another
* Application instance A processes to offset 4
* Application instance B comes up and fences application instance A
* Application instance B processes to offset 5
* Application instances A and B are idle for transaction.id.expiration.ms, transaction id expires on server
* Application instance A attempts to process offset 5 (since in its view, that is next) -- if we recover from invalid pid mapping, we can duplicate this processing
Thus, INVALID_PID_MAPPING should be fatal to the producer.
This is consistent with [KIP-1050: Consistent error handling for Transactions|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions] where errors that are fatal to the producer are in the "application recoverable" category. This is a grouping that indicates to the client that the producer needs to restart and recovery on the application side is necessary. KIP-1050 is approved so we are consistent with that decision.
h3.
> Convert INVALID_PRODUCER_ID_MAPPING from abortable error to fatal error
> -----------------------------------------------------------------------
>
> Key: KAFKA-18019
> URL: https://issues.apache.org/jira/browse/KAFKA-18019
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Ritika Reddy
> Assignee: Ritika Reddy
> Priority: Major
>
> Since we bump epoch on abort, we no longer need to call InitProducerId to fence requests. InitProducerId will only be called when the producer starts up to fence a previous instance.
> With this change, some other calls to InitProducerId were inspected including the call after receiving an InvalidPidMappingException. This exception was changed to abortable as part of [KIP-360: Improve reliability of idempotent/transactional producer|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820]. However, this change means that we can violate EOS guarantees. As an example:
> Consider an application that is copying data from one partition to another
> * Application instance A processes to offset 4
> * Application instance B comes up and fences application instance A
> * Application instance B processes to offset 5
> * Application instances A and B are idle for transaction.id.expiration.ms, transaction id expires on server
> * Application instance A attempts to process offset 5 (since in its view, that is next) -- if we recover from invalid pid mapping, we can duplicate this processing
> Thus, INVALID_PID_MAPPING should be fatal to the producer.
> This is consistent with [KIP-1050: Consistent error handling for Transactions|https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions] where errors that are fatal to the producer are in the "application recoverable" category. This is a grouping that indicates to the client that the producer needs to restart and recovery on the application side is necessary. KIP-1050 is approved so we are consistent with that decision.
> h3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)