Emulating a cosumer with maximum consumption rate and its synchronization with the producer work.

[Mazen Ezzeddine <ma...@etu.univ-cotedazur.fr>]

Dear all,

I am trying to run a Kafka experiment on Kubernetes where I would like to control the number of messages sent/produced  to the broker per unit time/interval, and the number of  messages being consumed by the consumer per unit time/interval.
Controlling the number of messages sent/produced  to the broker per unit time (according to a predetermined workload /trace) is easy. E.g., for a trace of 20 messages per 15 secs, and then 100 messages per next 15 secs, and finally, 45 msgs per the next 15 seconds…, the methodology is simple: send the 20 messages(in single batch), sleeping (the producer main thread) for 15 seconds,  followed  by sending the 100 messages, and sleeping the producer thread for 15 seconds and so on… I am sending the message in a single batch.

However, controlling the number of consumed messages per unit time (say 15 seconds) looks more complex. In fact, to emulate a consumer with maximum rate of 100 messages every 15 seconds interval, I am setting  max.poll.records=100 and sleeping the poll thread for 15 seconds. The issue is that I want to ensure that during the 15 seconds interval at the producer if this last produced say 150 messages, then during the next first 15 seconds interval at the consumer 100 (and not 150) messages are consumed (recall maximul.poll.records =100).
On the other hand, say during the 15 seconds interval at the producer if this last produced say 70 messages, then during the next first corresponding 15 seconds interval at the consumer exactly 70 messages are consumed; before sleeping the poll loop for 15 seconds, and continue so on.
===
Perhaps in simpler words: divide the producer and the consumer work  into s seconds interval, during the s seconds at the producer, this last produce p messages. The consumer is configured with c messages per second, and during the s seconds consumption interval (at the consumer) if p<c then exactly p messages are consumed, otherwise c messages are consumed.

Any hint please on the best way to achieve this experimental scenario? if any

Thanks so much for your time.

Re: Emulating a cosumer with maximum consumption rate and its synchronization with the producer work.

[Nikita Kretov <kr...@gmail.com>]

Hello! Interesting question, first things first I don't know the answer)
But there are quotation mechanism at broker side of this equation. Maybe 
it's possible to implement custom quota mechanism that will use message 
per sec metric instead of bytes per sec.

On 5/31/21 6:41 PM, Mazen Ezzeddine wrote:
> Dear all,
> 
> I am trying to run a Kafka experiment on Kubernetes where I would like to control the number of messages sent/produced  to the broker per unit time/interval, and the number of  messages being consumed by the consumer per unit time/interval.
> Controlling the number of messages sent/produced  to the broker per unit time (according to a predetermined workload /trace) is easy. E.g., for a trace of 20 messages per 15 secs, and then 100 messages per next 15 secs, and finally, 45 msgs per the next 15 seconds…, the methodology is simple: send the 20 messages(in single batch), sleeping (the producer main thread) for 15 seconds,  followed  by sending the 100 messages, and sleeping the producer thread for 15 seconds and so on… I am sending the message in a single batch.
> 
> However, controlling the number of consumed messages per unit time (say 15 seconds) looks more complex. In fact, to emulate a consumer with maximum rate of 100 messages every 15 seconds interval, I am setting  max.poll.records=100 and sleeping the poll thread for 15 seconds. The issue is that I want to ensure that during the 15 seconds interval at the producer if this last produced say 150 messages, then during the next first 15 seconds interval at the consumer 100 (and not 150) messages are consumed (recall maximul.poll.records =100).
> On the other hand, say during the 15 seconds interval at the producer if this last produced say 70 messages, then during the next first corresponding 15 seconds interval at the consumer exactly 70 messages are consumed; before sleeping the poll loop for 15 seconds, and continue so on.
> ===
> Perhaps in simpler words: divide the producer and the consumer work  into s seconds interval, during the s seconds at the producer, this last produce p messages. The consumer is configured with c messages per second, and during the s seconds consumption interval (at the consumer) if p<c then exactly p messages are consumed, otherwise c messages are consumed.
> 
> Any hint please on the best way to achieve this experimental scenario? if any
> 
> Thanks so much for your time.
>