You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Behrang Saeedzadeh <be...@gmail.com> on 2017/08/25 00:53:11 UTC

Does Kafka Connect have a built-in, connector-agnostic way to throttle its connection to a Kafka Cluster?

Hi,

As it can be seen in this diagram:
https://www.planttext.com/plantuml/img/ZP7BRi8m44Nt_eehi-O7ka1KYSI221QXOjFDa4p1meoHFH5yVXqUaNXKgjVddhcwdYQbgDt16wDEFSFEwMTFA888DuenLhiOLBsJhJNcc67Nk2E93gJladAqe8HvKRr5hh8Ridn4IaxgXDG5kG1blJ9c4ywmLMZ8Oy5w2d5lCQhSTgUpo2m6-VsZ2qkBhBhv2X-wzkwfoH_2twAdnY_OkzR3jNmQuqaVn0U-4YTyhckKBlQBtfVz5igyCBS9Ecni-b3ENbQOPcF-cby0

I have a Kafka Connect cluster that resides in AWS that consumes messages
from a Kafka cluster that resides in an on-prem network. There's a sink
connector on the Kafka Connect cluster that writes messages to a particular
destination.

The bandwidth between on-prem to AWS is limited and shared between many
services in the organisation.

Is there a way to configure Kafka Connect to make sure the connector does
not read more than, say, 1MB/s from Kafka to ensure it does not saturate
the bandwidth?

Thanks,
Behrang
-- 
Best regards,
Behrang Saeedzadeh