[jira] [Created] (KAFKA-7636) Allow consumer to update maxPollRecords value

["Kcirtap Seven (JIRA)" <ji...@apache.org>]

Kcirtap Seven created KAFKA-7636:
------------------------------------

             Summary: Allow consumer to update maxPollRecords value
                 Key: KAFKA-7636
                 URL: https://issues.apache.org/jira/browse/KAFKA-7636
             Project: Kafka
          Issue Type: Improvement
          Components: clients
    Affects Versions: 2.0.1
            Reporter: Kcirtap Seven


Hi,

We have two use cases where we would need to change the max.poll.records parameter on the fly :

1. We offer a REST API to get 'feedbacks'. This API takes into account a parameter 'count'. 
The system was previously based on cassandra. It is now based on kafka and 'feedbacks' are stored into kafka topics. 
To be compliant with the legacy interface contract, we would like to be able to change the max.poll.records on the fly to take into account this 'count' parameter. 

2. We receive 'notification requests' related to a 'sender' via a REST API. We store those requests into topics (by sender). 
Each sender is associated with a weight. Here is the algorithm that process the requests :

    1. At each iteration, we process at max n records (n configurable) for the whole bunch of requests. For this example, let's say 100. 
    2. We compute the max poll records for each sender. Let's say we have 3 senders with the following weight 2, 1, 1. Hence 50 records max for the first one, 25 for the others two. 
    3. We consume the topics one after the other. We would like to reallocate some capacity to remaining consumers if the max.poll.records is not reached for the current consumer. Let'say at each iteration we make the following synchronous calls :
            sender1Consumer.poll() with computed max.poll.records 50
            sender2Consumer.poll() with computed max.poll.records 25
            sender3Consumer.poll() with computed max.poll.records 25
       If the first call returns only 10 records, we would like to reallocate the 40 "spare" records to the other consumers, 20 for each for instance (or another strategy). We would make the following calls instead : 
            sender2Consumer.poll() with updated max.poll.records 45
            sender3Consumer.poll() with updated max.poll.records 45

 

For that requirement we also need to change the max.poll.records on the fly.


Regards,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)