[GitHub] [pulsar] Mrdigs opened a new issue #4629: Ability to publish and acknowledge messages atomically

[GitBox <gi...@apache.org>]

Mrdigs opened a new issue #4629: Ability to publish and acknowledge messages atomically
URL: https://github.com/apache/pulsar/issues/4629
 
 
   I have a consumer that transforms messages consumed on one topic and produces messages on another topic. Clearly I cannot acknowledge the consumed message until I have published the transformed message.
   
   The issue I have is in how to handle a situation where for message cannot be acknowledged due to some spurious situation where the produced message has already been sent, because if the acknowledgement fails them the message will be redelivered despite the fact that it has been consumed.
   
   I appreciate that Pulsar is designed to run in an HA environment but we all know that High Availability is not Total Availability, therefore the scenario where the message cannot be acknowledged after the transformed message is possible.
   
   How can I can I handle that possibility? As a minimum, I would need to be able to publish and acknowledge atomically, but what if I needed to publish to more than one topic before acknowledging? In that scenario, there could be a failure between the two message publishes, meaning that one message has been published, the second has not, and the consumed message cannot be acknowledged either.
   
   Surely, for a 100% fault tolerant system, there must be a transactional ability whereby a producer and a consumer can enlist in a single transaction and commit when all operations are successful? 
   
   This seems to be a fundamental feature requirement  of a system like this?
   
   I'm sorry if I've missed anything in the documentation that covers this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services