Slack digest for #general - 2018-02-24

[Apache Pulsar Slack <ap...@gmail.com>]

2018-02-23 10:43:31 UTC - Till Rathschlag: @Till Rathschlag has joined the channel
----
2018-02-23 10:55:27 UTC - Till Rathschlag: Hello everybody, I'm currently evaluating pulsar and I try to understand if it fits to the following usecase: I like to use pulsar (among others) as a task queue. I want my task producer to generate as many jobs as the consumers can work on, so I need some kind of communication consumers -> producer. I tried to build this with acknowledging but noticed that this is only propagated to pulsar and not back to the producer. So my question is, how would I do this? I thought about the following:
- Provide some other topic for job acknowledging
- Monitor the ack-ratio from the producer service
Is pulsar the right tool for this? I would be glad if someone can share their experience with this. Thanks in advance!
----
2018-02-23 16:58:33 UTC - Matteo Merli: @Till Rathschlag The primary function of a messaging system is to decouple the producers and the consumer and that’s way we don’t have correlation of consumers acks to producer :slightly_smiling_face:

However, if you’re not requiring exact precision, you can try using backlog quota to stop the producer. 
You can configure a very low quota (eg: 10MB or 1MB…) and the default action is to block the producers when the consumers accumulate that amount of “backlog” in the queue. 

I’m saying it’s not precise because the check for quota is only done periodically in background (every 1min by default I think) for efficiency reasons, so a user can go a bit over quota before getting stopped. 

If you need a more finer control, you could use a 2nd topic. For example: 
 * Consumer gets a message, process it
 * Consumer sends confirmation on the 2nd topic (referring to a particular msgId for 1st topic)
 * Consumer acks the message

Producer can do a kind of “semaphore” limiting the number of “in-processing” messages, by waiting for confirmations on the 2nd topic. This could work even if there are multiple producers, because you can ignore msg Ids that were published by other producers
----