You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gabriel Machado <gm...@gmail.com> on 2017/08/07 12:33:56 UTC
Avoid jvm swapping
Hi,
I have a 3 nodes cluster with 18 GB RAM and 2 GB swap.
Each node have the following JVMs (Xms=Xmx) :
- Zookeeper 2GB
- Kafka 12 GB
- Kafka mirror-maker DCa 1 GB
- Kafka mirror-maker DCb 1 GB
All th JVMs consume 16 GB. It leaves 2 GB for the OS (debian jessie 64
bits).
Why i have no swap free on these virtual machines ?
#free -m
total used free shared buffers cached
Mem: 18105 17940 164 0 38 6666
-/+ buffers/cache: 11235 6869
Swap: 2047 2045 2
I've read i should avoid jvm swapping.
What is the best way to do that ?
- modify swapiness threshold
- unmount all swap partition
- force the jvm to stay in memory with mlockall (
https://github.com/LucidWorks/mlockall-agent)
- Other solution
Gabriel.
Re: Avoid jvm swapping
Posted by Todd Palino <tp...@gmail.com>.
In production, you probably want to avoid stacking up the applications like
this. There’s a number of reasons:
1) Kafka’s performance is significantly increased by other applications not
polluting the OS page cache
2) Zookeeper has specific performance requirements - among them are a
dedicated disk for transaction logs that it can sequentially write to
3) Mirror maker chews up a lot of CPU and memory with decompression and
recompression of messages
Particular sizing of your systems is going to be dependent on the amount of
data you are moving around, but at the very least I would recommend that
your Kafka brokers, Zookeeper ensemble, and mirror makers be on separate
systems (stacking up the mirror makers on a common system is fine,
however). The Kafka brokers will need CPU and memory, and of course decent
storage to meet your retention and performance requirements. ZK needs a bit
of memory, and very good disk for the transaction logs, but it’s CPU
requirements are pretty light. Mirror maker needs CPU and memory, but it
has no real need of disk performance at all.
Sizing the brokers, you can probably get away with 3 or 4 GB of heap (this
is based on my experience running really large clusters at LinkedIn - even
at that heap size we were good for a long time), using G1 garbage
collection. The guidelines in the Kafka documentation for this are the ones
that I have developed over the last few years here. Reserve the rest of the
memory for the OS to manage - buffers and cache is your friend.
-Todd
On Mon, Aug 7, 2017 at 11:06 AM, Gabriel Machado <gm...@gmail.com>
wrote:
> Thanks Todd, i will set swapiness to 1.
>
> Theses machines will be the future production cluster for our main
> datacenter . We have 2 remote datacenters.
> Kafka will bufferize logs and elasticsearch will index its.
>
> Is it a bad practice to have all these JVMs on the same virtual machine ?
> What do you recommend (number of machines, quantity of GB, CPU...) ? For
> the moment, each node has 4 vcpu.
>
> Gabriel.
>
> 2017-08-07 15:45 GMT+02:00 Todd Palino <tp...@gmail.com>:
>
> > To avoid swap you should set swappiness to 1, not 0. 1 is a request
> (don't
> > swap if avoidable) whereas 0 is a demand (processes will be killed as OOM
> > instead of swapping.
> >
> > However, I'm wondering why you are running such large heaps. Most of the
> ZK
> > heap is used for storage of the data in memory, and it's obvious from
> your
> > setup that this is a development instance. So if ZK is only being used
> for
> > that Kafka cluster you are testing, you can go with a smaller heap.
> >
> > Also, for what reason are you running a 12 GB heap for Kafka? Even our
> > largest production clusters at LinkedIn are using a heap size of 6 GB
> right
> > now. You want to leave memory open for the OS to use for buffers and
> cache
> > in order to get better performance from consumers. You can see from that
> > output that it's trying to.
> >
> > It really looks like you're just overloading your system. In which case
> > swapping is to be expected.
> >
> > -Todd
> >
> >
> >
> > On Aug 7, 2017 8:34 AM, "Gabriel Machado" <gm...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I have a 3 nodes cluster with 18 GB RAM and 2 GB swap.
> > Each node have the following JVMs (Xms=Xmx) :
> > - Zookeeper 2GB
> > - Kafka 12 GB
> > - Kafka mirror-maker DCa 1 GB
> > - Kafka mirror-maker DCb 1 GB
> >
> > All th JVMs consume 16 GB. It leaves 2 GB for the OS (debian jessie 64
> > bits).
> > Why i have no swap free on these virtual machines ?
> >
> > #free -m
> > total used free shared buffers cached
> > Mem: 18105 17940 164 0 38 6666
> > -/+ buffers/cache: 11235 6869
> > Swap: 2047 2045 2
> >
> >
> > I've read i should avoid jvm swapping.
> > What is the best way to do that ?
> > - modify swapiness threshold
> > - unmount all swap partition
> > - force the jvm to stay in memory with mlockall (
> > https://github.com/LucidWorks/mlockall-agent)
> > - Other solution
> >
> > Gabriel.
> >
>
--
*Todd Palino*
Senior Staff Engineer, Site Reliability
Data Infrastructure Streaming
linkedin.com/in/toddpalino
Re: Avoid jvm swapping
Posted by Gabriel Machado <gm...@gmail.com>.
Thanks Todd, i will set swapiness to 1.
Theses machines will be the future production cluster for our main
datacenter . We have 2 remote datacenters.
Kafka will bufferize logs and elasticsearch will index its.
Is it a bad practice to have all these JVMs on the same virtual machine ?
What do you recommend (number of machines, quantity of GB, CPU...) ? For
the moment, each node has 4 vcpu.
Gabriel.
2017-08-07 15:45 GMT+02:00 Todd Palino <tp...@gmail.com>:
> To avoid swap you should set swappiness to 1, not 0. 1 is a request (don't
> swap if avoidable) whereas 0 is a demand (processes will be killed as OOM
> instead of swapping.
>
> However, I'm wondering why you are running such large heaps. Most of the ZK
> heap is used for storage of the data in memory, and it's obvious from your
> setup that this is a development instance. So if ZK is only being used for
> that Kafka cluster you are testing, you can go with a smaller heap.
>
> Also, for what reason are you running a 12 GB heap for Kafka? Even our
> largest production clusters at LinkedIn are using a heap size of 6 GB right
> now. You want to leave memory open for the OS to use for buffers and cache
> in order to get better performance from consumers. You can see from that
> output that it's trying to.
>
> It really looks like you're just overloading your system. In which case
> swapping is to be expected.
>
> -Todd
>
>
>
> On Aug 7, 2017 8:34 AM, "Gabriel Machado" <gm...@gmail.com> wrote:
>
> Hi,
>
> I have a 3 nodes cluster with 18 GB RAM and 2 GB swap.
> Each node have the following JVMs (Xms=Xmx) :
> - Zookeeper 2GB
> - Kafka 12 GB
> - Kafka mirror-maker DCa 1 GB
> - Kafka mirror-maker DCb 1 GB
>
> All th JVMs consume 16 GB. It leaves 2 GB for the OS (debian jessie 64
> bits).
> Why i have no swap free on these virtual machines ?
>
> #free -m
> total used free shared buffers cached
> Mem: 18105 17940 164 0 38 6666
> -/+ buffers/cache: 11235 6869
> Swap: 2047 2045 2
>
>
> I've read i should avoid jvm swapping.
> What is the best way to do that ?
> - modify swapiness threshold
> - unmount all swap partition
> - force the jvm to stay in memory with mlockall (
> https://github.com/LucidWorks/mlockall-agent)
> - Other solution
>
> Gabriel.
>
Re: Avoid jvm swapping
Posted by Todd Palino <tp...@gmail.com>.
To avoid swap you should set swappiness to 1, not 0. 1 is a request (don't
swap if avoidable) whereas 0 is a demand (processes will be killed as OOM
instead of swapping.
However, I'm wondering why you are running such large heaps. Most of the ZK
heap is used for storage of the data in memory, and it's obvious from your
setup that this is a development instance. So if ZK is only being used for
that Kafka cluster you are testing, you can go with a smaller heap.
Also, for what reason are you running a 12 GB heap for Kafka? Even our
largest production clusters at LinkedIn are using a heap size of 6 GB right
now. You want to leave memory open for the OS to use for buffers and cache
in order to get better performance from consumers. You can see from that
output that it's trying to.
It really looks like you're just overloading your system. In which case
swapping is to be expected.
-Todd
On Aug 7, 2017 8:34 AM, "Gabriel Machado" <gm...@gmail.com> wrote:
Hi,
I have a 3 nodes cluster with 18 GB RAM and 2 GB swap.
Each node have the following JVMs (Xms=Xmx) :
- Zookeeper 2GB
- Kafka 12 GB
- Kafka mirror-maker DCa 1 GB
- Kafka mirror-maker DCb 1 GB
All th JVMs consume 16 GB. It leaves 2 GB for the OS (debian jessie 64
bits).
Why i have no swap free on these virtual machines ?
#free -m
total used free shared buffers cached
Mem: 18105 17940 164 0 38 6666
-/+ buffers/cache: 11235 6869
Swap: 2047 2045 2
I've read i should avoid jvm swapping.
What is the best way to do that ?
- modify swapiness threshold
- unmount all swap partition
- force the jvm to stay in memory with mlockall (
https://github.com/LucidWorks/mlockall-agent)
- Other solution
Gabriel.