Kafka back-pressure | Proposal for dynamic config

[Vipul Goyal <vi...@gmail.com>]

Hi Team,

I am looking for an effective back pressure solution in Kafka Please find
below my use case and detail.

Use case:
I need to run “some execution”  when receiving a Kafka record.  Some
execution could be an external API call, And it is possible that sometimes
this API may not perform well. Considering we already have API timeout
configured.
Now, let’s say there is some degradation in external API, and batch
processing will start taking much time and overall it may breach
max.poll.internal.ms and hence rebalancing. This would have a ripple effect
on other consumers in the same group.

As-is:
Currently, Kafka consumer provide the capability to pause/resume, but it
may not be very effective to back-pressure the flow.
1. Ideally We may need to just slow down, not exactly pause the consumption
itself.
2. If we pause the consumer, then we would have to remember to resume it.


*Proposal?*
To be able to auto recover from this, we can have some control algorithm
(ex PID) in place, which will adjust *max.poll.records* dynamically and we
can avoid rebalancing. And I see that PID (proportional integral
derivative) is being referenced in many other places as well like:
PIDRateEstimator
<https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/scheduler/rate/PIDRateEstimator.scala>
,
tuning-spark-back-pressure-by-simulation
<https://richardstartin.github.io/posts/tuning-spark-back-pressure-by-simulation>

kafka-spark-consumer <https://github.com/dibbhatt/kafka-spark-consumer>.

Currently, the Kafka consumer doesn't allow dynamic change of the max poll
records setting, But the proposal is to allow passing in this config
dynamically.

I am not an expert and may be thinking in an orthogonal direction. Please
advice.

Regards,
Vipul