You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "prasathsekar (via GitHub)" <gi...@apache.org> on 2024/05/29 04:28:43 UTC

[I] Duplicate message created using partitioned topic with flink [pulsar]

prasathsekar opened a new issue, #22795:
URL: https://github.com/apache/pulsar/issues/22795

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Read release policy
   
   - [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   
   ### Version
   
   3.2.3
   
   ### Minimal reproduce step
   
   Created a partitioned topic in Apache Pulsar and using Flink to consume it. Flink job has a parallelism of 10. If the JOB is restarted getting duplicate(same) messages
   
   ### What did you expect to see?
   
   Need only the unique message
   
   ### What did you see instead?
   
   While JOB restart getting same message 5-10 times
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Duplicate message created using partitioned topic with flink [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #22795:
URL: https://github.com/apache/pulsar/issues/22795#issuecomment-6381511447

   Thanks for the report @prasathsekar. There isn't enough here to tell whether this is a Pulsar problem. The Flink Pulsar source is maintained in https://github.com/apache/flink-connector-pulsar, and after a job restart it replays from the last completed checkpoint, so messages read after that checkpoint arrive again unless the pipeline is set up for exactly-once end to end. If you still see duplicates beyond that on a currently supported release (4.0.x, 4.2.x or 5.0.x), please share a reproduction: the Flink and connector versions, the source configuration (subscription type, delivery guarantee, checkpoint settings), whether the producer has deduplication enabled, and the message IDs of the duplicates. If there's no update, we'll close this issue in a few weeks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org