You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gérald Quintana <ge...@gmail.com> on 2020/06/19 12:56:12 UTC
Broker thread pool sizing
Hello,
How do you size Kafka broker thread pools, in particular num.io.threads (8
by default) and num.network.threads (3 by default) depending on the number
of CPU cores available on the host?
Regards,
Gérald
Re: Broker thread pool sizing
Posted by Gérald Quintana <ge...@gmail.com>.
Thank you very much Ricardo,
It's crystal clear.
Gérald
Le ven. 19 juin 2020 à 18:01, Ricardo Ferreira <ri...@riferrei.com> a
écrit :
> Gérald,
>
> Typically you should set the `num.io.threads` to something greater than
> the # of disks since data hits the page cache and the disk. Using the
> default of 8 when you have a JBOD of 12 attached volumes would cause an
> increase of CPU context switching, for example.
>
> `num.network.threads` is usually fine since most machines have 1 or 2
> NIC's attached at tops and then the rule of *N + 1* suits the case.
> However, you should double that if TLS is enabled in the broker.
>
> Thanks,
>
> -- Ricardo
> On 6/19/20 8:56 AM, Gérald Quintana wrote:
>
> Hello,
>
> How do you size Kafka broker thread pools, in particular num.io.threads (8
> by default) and num.network.threads (3 by default) depending on the number
> of CPU cores available on the host?
>
> Regards,
> Gérald
>
>
>
Re: Broker thread pool sizing
Posted by sunil chaudhari <su...@gmail.com>.
This is awesome. Thanks Ricardo.
On Fri, 19 Jun 2020 at 9:31 PM, Ricardo Ferreira <ri...@riferrei.com>
wrote:
> Gérald,
>
> Typically you should set the `num.io.threads` to something greater than
> the # of disks since data hits the page cache and the disk. Using the
> default of 8 when you have a JBOD of 12 attached volumes would cause an
> increase of CPU context switching, for example.
>
> `num.network.threads` is usually fine since most machines have 1 or 2
> NIC's attached at tops and then the rule of *N + 1* suits the case.
> However, you should double that if TLS is enabled in the broker.
>
> Thanks,
>
> -- Ricardo
>
> On 6/19/20 8:56 AM, Gérald Quintana wrote:
> > Hello,
> >
> > How do you size Kafka broker thread pools, in particular num.io.threads
> (8
> > by default) and num.network.threads (3 by default) depending on the
> number
> > of CPU cores available on the host?
> >
> > Regards,
> > Gérald
> >
>
Re: Broker thread pool sizing
Posted by Ricardo Ferreira <ri...@riferrei.com>.
Gérald,
Typically you should set the `num.io.threads` to something greater than
the # of disks since data hits the page cache and the disk. Using the
default of 8 when you have a JBOD of 12 attached volumes would cause an
increase of CPU context switching, for example.
`num.network.threads` is usually fine since most machines have 1 or 2
NIC's attached at tops and then the rule of *N + 1* suits the case.
However, you should double that if TLS is enabled in the broker.
Thanks,
-- Ricardo
On 6/19/20 8:56 AM, Gérald Quintana wrote:
> Hello,
>
> How do you size Kafka broker thread pools, in particular num.io.threads (8
> by default) and num.network.threads (3 by default) depending on the number
> of CPU cores available on the host?
>
> Regards,
> Gérald
>