[GitHub] [pulsar] sleungtoast opened a new issue #5218: Enabling Deduplication may drop messages

[GitBox <gi...@apache.org>]

sleungtoast opened a new issue #5218: Enabling Deduplication may drop messages
URL: https://github.com/apache/pulsar/issues/5218
 
 
   **Describe the bug**
   Since deduplication drops messages based on the the largest sequence id recorded pre-persist, if there’s an error persisting to BK, a retry attempt will just be “deduplicated” with no message ever getting persisted.
   <img width="801" alt="dedupe_fail_case" src="https://user-images.githubusercontent.com/7429691/65181039-2a068f80-da2c-11e9-9d24-550ab3c0a220.png">
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. Client connects successfully, producer send retry enabled (i.e. sendTimeoutMs at default of 30s)
   2. Cause some BK failures by shut down of some nodes or a rolling restart
   3. While that happens, Client reports `Received send error from server` (note it's send error, not just a normal connection error). One example of the several resulting errors are `PersistenceError : org.apache.bookkeeper.mledger.ManagedLedgerException: Waiting for new ledger creation to complete`
   3. Looking at broker logs, will see `Message has been dropped for non-persistent topic producer-id ...` even though the first attempt was unsuccessful
   4. Message was lost due to dedupe logic counting 2nd retry attempt as duplicate.
   
   **Expected behavior**
   Messages should not be dropped
   
   **Desktop (please complete the following information):**
    - OS: OSX, AWS Linux
   
   **Additional context**
   Instructions for dedupe at this link, but the same problem can occur with those settings. Also, does setting sendTimeoutMs to 0 make it try forever, or does it drop the message on first fail? https://pulsar.apache.org/docs/en/cookbooks-deduplication/#message-deduplication-and-pulsar-clients-clients
   Logic for dedupe in `MessageDeduplication.java`
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services