Kafka streams consumer/producer throttling

[Andrey Dyachkov <an...@gmail.com>]

Hello,

Could you tell me if there is a way to throttle kafka streams by client id?
I’ve tried to set bandwidth limit for client id, which is specefied in
kafka stream config, but it does not work.
-- 

With great enthusiasm,
Andrey

Re: Kafka streams consumer/producer throttling

[Guozhang Wang <wa...@gmail.com>]

Hi Andrey,

You're right that Streams does not allow users to directly set client ids
for the embedded producer / consumer in order to maintain uniqueness. If
you take a look at `StreamsConfig#getProducerConfigs` for example you'll
see at the end we always override the client id configs.

Right now you'll have to use the constructed client ids to set the configs,
as well as acls. We have a JIRA open for adding a Streams role in those
tools for easy operations but atm manual settings are unfortunately not
avoidable.

As for wildcards, for acls there are support for prefix wildcard but I
think we do not have similar support for general config settings for entity
names yet.


Guozhang


On Tue, Nov 20, 2018 at 12:51 AM Andrey Dyachkov <an...@gmail.com>
wrote:

> Hi Guozhang,
>
> I set client id like that:
> propsWithAppId.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
> propsWithAppId.put(StreamsConfig.CLIENT_ID_CONFIG, appId);
> propsWithAppId.put(ProducerConfig.CLIENT_ID_CONFIG, appId);
> propsWithAppId.put(ConsumerConfig.CLIENT_ID_CONFIG, appId);
>
> propsWithAppId.put(StreamsConfig.producerPrefix(ProducerConfig.CLIENT_ID_CONFIG),
> appId);
>
> propsWithAppId.put(StreamsConfig.consumerPrefix(ConsumerConfig.CLIENT_ID_CONFIG),
> appId);
>
> after I tried to set it in 2 different ways neither of them worked:
> AdminUtils.changeClientIdConfig(zkUtils, clientId, clientProps); // i
> know it is deprecated
>  bin/kafka-configs.sh  --zookeeper zk-ips --alter --add-config
> 'producer_byte_rate=1048576,consumer_byte_rate=1048576' --entity-type
> clients --entity-name appId
>
> I checked the source code of the kstreams. The client id is actually
> changed by kstreams, that's why setting it does not have any effect.
> It forms client id as: clientId + thread name + (consumer | producer |
> restore-consumer)
>
> I have not found in the docs if it is possible to use wildcard when
> setting client id bandwidth threshold.
>
> On Sat, 17 Nov 2018 at 01:23, Guozhang Wang <wa...@gmail.com> wrote:
> >
> > Hello Andrey,
> >
> > Could you provide a bit more information on how you set the bandwidth
> based
> > on client id? Sharing some code snippet would even better for me to
> > understand your encountered issue.
> >
> > Guozhang
> >
> > On Fri, Nov 9, 2018 at 7:11 AM Andrey Dyachkov <
> andrey.dyachkov@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Could you tell me if there is a way to throttle kafka streams by
> client id?
> > > I’ve tried to set bandwidth limit for client id, which is specefied in
> > > kafka stream config, but it does not work.
> > > --
> > >
> > > With great enthusiasm,
> > > Andrey
> > >
> >
> >
> > --
> > -- Guozhang
>
>
>
> --
> Thanks,
> Andrey
>


-- 
-- Guozhang

Re: Kafka streams consumer/producer throttling

[Andrey Dyachkov <an...@gmail.com>]

Hi Guozhang,

I set client id like that:
propsWithAppId.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
propsWithAppId.put(StreamsConfig.CLIENT_ID_CONFIG, appId);
propsWithAppId.put(ProducerConfig.CLIENT_ID_CONFIG, appId);
propsWithAppId.put(ConsumerConfig.CLIENT_ID_CONFIG, appId);
propsWithAppId.put(StreamsConfig.producerPrefix(ProducerConfig.CLIENT_ID_CONFIG),
appId);
propsWithAppId.put(StreamsConfig.consumerPrefix(ConsumerConfig.CLIENT_ID_CONFIG),
appId);

after I tried to set it in 2 different ways neither of them worked:
AdminUtils.changeClientIdConfig(zkUtils, clientId, clientProps); // i
know it is deprecated
 bin/kafka-configs.sh  --zookeeper zk-ips --alter --add-config
'producer_byte_rate=1048576,consumer_byte_rate=1048576' --entity-type
clients --entity-name appId

I checked the source code of the kstreams. The client id is actually
changed by kstreams, that's why setting it does not have any effect.
It forms client id as: clientId + thread name + (consumer | producer |
restore-consumer)

I have not found in the docs if it is possible to use wildcard when
setting client id bandwidth threshold.

On Sat, 17 Nov 2018 at 01:23, Guozhang Wang <wa...@gmail.com> wrote:
>
> Hello Andrey,
>
> Could you provide a bit more information on how you set the bandwidth based
> on client id? Sharing some code snippet would even better for me to
> understand your encountered issue.
>
> Guozhang
>
> On Fri, Nov 9, 2018 at 7:11 AM Andrey Dyachkov <an...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Could you tell me if there is a way to throttle kafka streams by client id?
> > I’ve tried to set bandwidth limit for client id, which is specefied in
> > kafka stream config, but it does not work.
> > --
> >
> > With great enthusiasm,
> > Andrey
> >
>
>
> --
> -- Guozhang



-- 
Thanks,
Andrey

Re: Kafka streams consumer/producer throttling

[Guozhang Wang <wa...@gmail.com>]

Hello Andrey,

Could you provide a bit more information on how you set the bandwidth based
on client id? Sharing some code snippet would even better for me to
understand your encountered issue.

Guozhang

On Fri, Nov 9, 2018 at 7:11 AM Andrey Dyachkov <an...@gmail.com>
wrote:

> Hello,
>
> Could you tell me if there is a way to throttle kafka streams by client id?
> I’ve tried to set bandwidth limit for client id, which is specefied in
> kafka stream config, but it does not work.
> --
>
> With great enthusiasm,
> Andrey
>


-- 
-- Guozhang