You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Shivam Sharma <28...@gmail.com> on 2018/01/09 11:49:50 UTC

Need broker side property which can restrict a fetch.size

Hi,

We are having Kafka cluster(0.11.0.0) of 3 Machines. When we increase
*max.fetch.size.per.partition* to around 50 MB for topic having 48
partition, our Kafka Broker is getting OOM.

As our cluster has many consumers, so restricting fetch size on consumer
side is very difficult which is continuously leading to OOM like issues as
consumer has power to do this.

Anything can be done on broker side to avoid these issues by restricting
clients.

Thanks

-- 
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*

Re: Need broker side property which can restrict a fetch.size

Posted by Shivam Sharma <28...@gmail.com>.

Thanks Srinath, We will try to do quota management.

On Wed, Jan 10, 2018 at 1:18 PM, Srinath C <sr...@gmail.com> wrote:

> Hi Shivam,
>
> You can configure quotas per user/client.
>
> [1] https://kafka.apache.org/documentation/#design_quotas
>
> Regards,
> Srinath
>
> On Jan 9, 2018 5:19 PM, "Shivam Sharma" <28...@gmail.com> wrote:
>
> > Hi,
> >
> > We are having Kafka cluster(0.11.0.0) of 3 Machines. When we increase
> > *max.fetch.size.per.partition* to around 50 MB for topic having 48
> > partition, our Kafka Broker is getting OOM.
> >
> > As our cluster has many consumers, so restricting fetch size on consumer
> > side is very difficult which is continuously leading to OOM like issues
> as
> > consumer has power to do this.
> >
> > Anything can be done on broker side to avoid these issues by restricting
> > clients.
> >
> > Thanks
> >
> > --
> > Shivam Sharma
> > Data Engineer @ Goibibo
> > Indian Institute Of Information Technology, Design and Manufacturing
> > Jabalpur
> > Mobile No- (+91) 8882114744
> > Email:- 28shivamsharma@gmail.com
> > LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> > <https://www.linkedin.com/in/28shivamsharma>*
> >
>



-- 
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Information Technology, Design and Manufacturing
Jabalpur
Mobile No- (+91) 8882114744
Email:- 28shivamsharma@gmail.com
LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
<https://www.linkedin.com/in/28shivamsharma>*

Re: Need broker side property which can restrict a fetch.size

Posted by Srinath C <sr...@gmail.com>.

Hi Shivam,

You can configure quotas per user/client.

[1] https://kafka.apache.org/documentation/#design_quotas

Regards,
Srinath

On Jan 9, 2018 5:19 PM, "Shivam Sharma" <28...@gmail.com> wrote:

> Hi,
>
> We are having Kafka cluster(0.11.0.0) of 3 Machines. When we increase
> *max.fetch.size.per.partition* to around 50 MB for topic having 48
> partition, our Kafka Broker is getting OOM.
>
> As our cluster has many consumers, so restricting fetch size on consumer
> side is very difficult which is continuously leading to OOM like issues as
> consumer has power to do this.
>
> Anything can be done on broker side to avoid these issues by restricting
> clients.
>
> Thanks
>
> --
> Shivam Sharma
> Data Engineer @ Goibibo
> Indian Institute Of Information Technology, Design and Manufacturing
> Jabalpur
> Mobile No- (+91) 8882114744
> Email:- 28shivamsharma@gmail.com
> LinkedIn:-*https://www.linkedin.com/in/28shivamsharma
> <https://www.linkedin.com/in/28shivamsharma>*
>