[I] [Bug] Consumer might not be able to consume messages after failover to a new cluster [pulsar]

["BewareMyPower (via GitHub)" <gi...@apache.org>]

BewareMyPower opened a new issue, #25291:
URL: https://github.com/apache/pulsar/issues/25291

   ### Search before reporting
   
   - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Read release policy
   
   - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   
   ### User environment
   
   master branch
   
   ### Issue Description
   
   With `AutoClusterFailover`, a consumer is able to switch from cluster A to cluster B on the same topic. 
   
   https://github.com/apache/pulsar/blob/629ddebeec08438536a6b63f413c11caaf445b1f/pulsar-client/src/main/java/org/apache/pulsar/client/impl/AutoClusterFailover.java#L172-L173
   
   The client will update the service URL and recreate the lookup service. However, the consumer's internal states are not cleared, including:
   - The ACK grouping tracker, which contains some acknowledged message IDs
   - `lastDequedMessageId`, which represents the message ID of last message received
   
   These message IDs all come from **Cluster A**.
   
   Next time, when the consumer established connection to Cluster B, it will:
   - Filter out duplicated messages via `AcknowledgmentsGroupingTracker#isDuplicate`. If the acknowledged message ID is cumulative, it will compare the newly **received message's ID from Cluster B** with **the previous acknowledged message ID from Cluster A**
   - Set `startMessageId` to the **last message ID from Cluster A**, this message ID will be carried in **the Subscribe request to the topic on Cluster B**
   
   ### Error messages
   
   ```text
   n/a
   ```
   
   ### Reproducing the issue
   
   I did't try to reproduce it with Java client for now, but it can be reproduced with C++ client like: https://github.com/apache/pulsar-client-cpp/pull/541/changes/c6de067e3795b58eed558c03795e385fc8ee16a5#diff-36936d31d0cbc6547ff0eea6b2bc79bbffdc4de6d63223dd0b4e527e369059d1
   
   ### Additional information
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Consumer might not be able to consume messages after failover to a new cluster [pulsar]

["merlimat (via GitHub)" <gi...@apache.org>]

merlimat commented on issue #25291:
URL: https://github.com/apache/pulsar/issues/25291#issuecomment-6378757146

   Confirmed on master for the Java client. `AutoClusterFailover` switches clusters through `PulsarClientImpl.updateServiceUrl` (`pulsar-client/src/main/java/org/apache/pulsar/client/impl/PulsarClientImpl.java:1718`), which only replaces the lookup and closes connections. Consumers keep their cluster-A state. The ack grouping tracker's `lastCumulativeAck` is still used by `isDuplicate` (`pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java:130`, called from `ConsumerImpl.java:1508` and `:2004`), so messages from cluster B can be dropped as duplicates. For non-durable subscriptions and readers, `lastDequeuedMessageId` from cluster A becomes the `startMessageId` (`ConsumerImpl.java:1147`). The tracker is reset today only on seek or on a blue-green redirect (`ConsumerImpl.java:3611`). Likely fix: when the service URL changes, clear each consumer's ack tracker and its `lastDequeuedMessageId`/`startMessageId`, as the blue-green redirect p
 ath does and as pulsar-client-cpp#541 did for C++.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org