You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Theo Hultberg <th...@iconara.net> on 2014/09/01 11:52:18 UTC
Replication bandwidth double what is expected
Hi,
We're evaluating Kafka, and have a problem with it using more bandwidth
than we can explain. From what we can tell the replication uses at least
twice the bandwidth it should.
We have four producer nodes and three broker nodes. We have enabled 3x
replication, so each node will get a copy of all data in this setup. The
producers have Snappy compression enabled and send batches of 200 messages.
The messages are around 1 KiB each. The cluster runs using mostly default
configuration, and the Kafka version is 0.8.1.1.
When we run iftop on the broker nodes we see that each Kafka node receives
around 6-7 Mbit from each producer node (or around 25-30 Mbit in total),
but then sends around 50 Mbit to each other Kafka node (or 100 Mbit in
total). This is twice what we expected to see, and it seems to saturate the
bandwidth on our m1.xlarge machines. In other words, we expected the
incoming 25 Mbit to be amplified to 50 Mbit, not 100.
One thing that could explain it, and that we don't really know how to
verify, is that the inter-node communication is not compressed. We aren't
sure about what compression ratio we get on the incoming data, but 50%
sounds reasonable. Could this explain what we're seeing? Is there a
configuration property to enable compression on the replication traffic
that we've missed?
yours
Theo
Re: Replication bandwidth double what is expected
Posted by Neha Narkhede <ne...@gmail.com>.
Theo,
Wondering if changing those settings as suggested worked for you?
Thanks,
Neha
On Wed, Sep 3, 2014 at 12:51 AM, Theo Hultberg <th...@iconara.net> wrote:
> that makes sense. if the size of each fetch is small then compression won't
> do much, and that could very well explain the increase in bandwidth.
>
> we will try to change these settings and see what happens.
>
> thanks a lot for your help.
>
> T#
>
>
> On Tue, Sep 2, 2014 at 10:44 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hi Theo,
> >
> > You can try to set replica.fetch.min.bytes to some large number (default
> to
> > 1) and increase replica.fetch.wait.max.ms (default to 500) and see if
> that
> > helps. In general, with 4 fetchers and min.bytes to 1 the replicas would
> > effectively exchange many small packets over the wire.
> >
> > Guozhang
> >
> >
> > On Mon, Sep 1, 2014 at 11:06 PM, Theo Hultberg <th...@iconara.net> wrote:
> >
> > > Hi Guozhang,
> > >
> > > We're using the default on all of those, except num.replica.fetchers
> > which
> > > is set to 4.
> > >
> > > T#
> > >
> > >
> > > On Mon, Sep 1, 2014 at 9:41 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Hello Theo,
> > > >
> > > > What are the values for your "replica.fetch.max.bytes",
> > > > "replica.fetch.min.bytes", "replica.fetch.wait.max.ms" and
> > > > "num.replica.fetchers" configs?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Mon, Sep 1, 2014 at 2:52 AM, Theo Hultberg <th...@iconara.net>
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We're evaluating Kafka, and have a problem with it using more
> > bandwidth
> > > > > than we can explain. From what we can tell the replication uses at
> > > least
> > > > > twice the bandwidth it should.
> > > > >
> > > > > We have four producer nodes and three broker nodes. We have enabled
> > 3x
> > > > > replication, so each node will get a copy of all data in this
> setup.
> > > The
> > > > > producers have Snappy compression enabled and send batches of 200
> > > > messages.
> > > > > The messages are around 1 KiB each. The cluster runs using mostly
> > > default
> > > > > configuration, and the Kafka version is 0.8.1.1.
> > > > >
> > > > > When we run iftop on the broker nodes we see that each Kafka node
> > > > receives
> > > > > around 6-7 Mbit from each producer node (or around 25-30 Mbit in
> > > total),
> > > > > but then sends around 50 Mbit to each other Kafka node (or 100 Mbit
> > in
> > > > > total). This is twice what we expected to see, and it seems to
> > saturate
> > > > the
> > > > > bandwidth on our m1.xlarge machines. In other words, we expected
> the
> > > > > incoming 25 Mbit to be amplified to 50 Mbit, not 100.
> > > > >
> > > > > One thing that could explain it, and that we don't really know how
> to
> > > > > verify, is that the inter-node communication is not compressed. We
> > > aren't
> > > > > sure about what compression ratio we get on the incoming data, but
> > 50%
> > > > > sounds reasonable. Could this explain what we're seeing? Is there a
> > > > > configuration property to enable compression on the replication
> > traffic
> > > > > that we've missed?
> > > > >
> > > > > yours
> > > > > Theo
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: Replication bandwidth double what is expected
Posted by Theo Hultberg <th...@iconara.net>.
that makes sense. if the size of each fetch is small then compression won't
do much, and that could very well explain the increase in bandwidth.
we will try to change these settings and see what happens.
thanks a lot for your help.
T#
On Tue, Sep 2, 2014 at 10:44 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Theo,
>
> You can try to set replica.fetch.min.bytes to some large number (default to
> 1) and increase replica.fetch.wait.max.ms (default to 500) and see if that
> helps. In general, with 4 fetchers and min.bytes to 1 the replicas would
> effectively exchange many small packets over the wire.
>
> Guozhang
>
>
> On Mon, Sep 1, 2014 at 11:06 PM, Theo Hultberg <th...@iconara.net> wrote:
>
> > Hi Guozhang,
> >
> > We're using the default on all of those, except num.replica.fetchers
> which
> > is set to 4.
> >
> > T#
> >
> >
> > On Mon, Sep 1, 2014 at 9:41 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Hello Theo,
> > >
> > > What are the values for your "replica.fetch.max.bytes",
> > > "replica.fetch.min.bytes", "replica.fetch.wait.max.ms" and
> > > "num.replica.fetchers" configs?
> > >
> > > Guozhang
> > >
> > >
> > > On Mon, Sep 1, 2014 at 2:52 AM, Theo Hultberg <th...@iconara.net>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > We're evaluating Kafka, and have a problem with it using more
> bandwidth
> > > > than we can explain. From what we can tell the replication uses at
> > least
> > > > twice the bandwidth it should.
> > > >
> > > > We have four producer nodes and three broker nodes. We have enabled
> 3x
> > > > replication, so each node will get a copy of all data in this setup.
> > The
> > > > producers have Snappy compression enabled and send batches of 200
> > > messages.
> > > > The messages are around 1 KiB each. The cluster runs using mostly
> > default
> > > > configuration, and the Kafka version is 0.8.1.1.
> > > >
> > > > When we run iftop on the broker nodes we see that each Kafka node
> > > receives
> > > > around 6-7 Mbit from each producer node (or around 25-30 Mbit in
> > total),
> > > > but then sends around 50 Mbit to each other Kafka node (or 100 Mbit
> in
> > > > total). This is twice what we expected to see, and it seems to
> saturate
> > > the
> > > > bandwidth on our m1.xlarge machines. In other words, we expected the
> > > > incoming 25 Mbit to be amplified to 50 Mbit, not 100.
> > > >
> > > > One thing that could explain it, and that we don't really know how to
> > > > verify, is that the inter-node communication is not compressed. We
> > aren't
> > > > sure about what compression ratio we get on the incoming data, but
> 50%
> > > > sounds reasonable. Could this explain what we're seeing? Is there a
> > > > configuration property to enable compression on the replication
> traffic
> > > > that we've missed?
> > > >
> > > > yours
> > > > Theo
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Replication bandwidth double what is expected
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Theo,
You can try to set replica.fetch.min.bytes to some large number (default to
1) and increase replica.fetch.wait.max.ms (default to 500) and see if that
helps. In general, with 4 fetchers and min.bytes to 1 the replicas would
effectively exchange many small packets over the wire.
Guozhang
On Mon, Sep 1, 2014 at 11:06 PM, Theo Hultberg <th...@iconara.net> wrote:
> Hi Guozhang,
>
> We're using the default on all of those, except num.replica.fetchers which
> is set to 4.
>
> T#
>
>
> On Mon, Sep 1, 2014 at 9:41 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Hello Theo,
> >
> > What are the values for your "replica.fetch.max.bytes",
> > "replica.fetch.min.bytes", "replica.fetch.wait.max.ms" and
> > "num.replica.fetchers" configs?
> >
> > Guozhang
> >
> >
> > On Mon, Sep 1, 2014 at 2:52 AM, Theo Hultberg <th...@iconara.net> wrote:
> >
> > > Hi,
> > >
> > > We're evaluating Kafka, and have a problem with it using more bandwidth
> > > than we can explain. From what we can tell the replication uses at
> least
> > > twice the bandwidth it should.
> > >
> > > We have four producer nodes and three broker nodes. We have enabled 3x
> > > replication, so each node will get a copy of all data in this setup.
> The
> > > producers have Snappy compression enabled and send batches of 200
> > messages.
> > > The messages are around 1 KiB each. The cluster runs using mostly
> default
> > > configuration, and the Kafka version is 0.8.1.1.
> > >
> > > When we run iftop on the broker nodes we see that each Kafka node
> > receives
> > > around 6-7 Mbit from each producer node (or around 25-30 Mbit in
> total),
> > > but then sends around 50 Mbit to each other Kafka node (or 100 Mbit in
> > > total). This is twice what we expected to see, and it seems to saturate
> > the
> > > bandwidth on our m1.xlarge machines. In other words, we expected the
> > > incoming 25 Mbit to be amplified to 50 Mbit, not 100.
> > >
> > > One thing that could explain it, and that we don't really know how to
> > > verify, is that the inter-node communication is not compressed. We
> aren't
> > > sure about what compression ratio we get on the incoming data, but 50%
> > > sounds reasonable. Could this explain what we're seeing? Is there a
> > > configuration property to enable compression on the replication traffic
> > > that we've missed?
> > >
> > > yours
> > > Theo
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Replication bandwidth double what is expected
Posted by Theo Hultberg <th...@iconara.net>.
Hi Guozhang,
We're using the default on all of those, except num.replica.fetchers which
is set to 4.
T#
On Mon, Sep 1, 2014 at 9:41 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Theo,
>
> What are the values for your "replica.fetch.max.bytes",
> "replica.fetch.min.bytes", "replica.fetch.wait.max.ms" and
> "num.replica.fetchers" configs?
>
> Guozhang
>
>
> On Mon, Sep 1, 2014 at 2:52 AM, Theo Hultberg <th...@iconara.net> wrote:
>
> > Hi,
> >
> > We're evaluating Kafka, and have a problem with it using more bandwidth
> > than we can explain. From what we can tell the replication uses at least
> > twice the bandwidth it should.
> >
> > We have four producer nodes and three broker nodes. We have enabled 3x
> > replication, so each node will get a copy of all data in this setup. The
> > producers have Snappy compression enabled and send batches of 200
> messages.
> > The messages are around 1 KiB each. The cluster runs using mostly default
> > configuration, and the Kafka version is 0.8.1.1.
> >
> > When we run iftop on the broker nodes we see that each Kafka node
> receives
> > around 6-7 Mbit from each producer node (or around 25-30 Mbit in total),
> > but then sends around 50 Mbit to each other Kafka node (or 100 Mbit in
> > total). This is twice what we expected to see, and it seems to saturate
> the
> > bandwidth on our m1.xlarge machines. In other words, we expected the
> > incoming 25 Mbit to be amplified to 50 Mbit, not 100.
> >
> > One thing that could explain it, and that we don't really know how to
> > verify, is that the inter-node communication is not compressed. We aren't
> > sure about what compression ratio we get on the incoming data, but 50%
> > sounds reasonable. Could this explain what we're seeing? Is there a
> > configuration property to enable compression on the replication traffic
> > that we've missed?
> >
> > yours
> > Theo
> >
>
>
>
> --
> -- Guozhang
>
Re: Replication bandwidth double what is expected
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Theo,
What are the values for your "replica.fetch.max.bytes",
"replica.fetch.min.bytes", "replica.fetch.wait.max.ms" and
"num.replica.fetchers" configs?
Guozhang
On Mon, Sep 1, 2014 at 2:52 AM, Theo Hultberg <th...@iconara.net> wrote:
> Hi,
>
> We're evaluating Kafka, and have a problem with it using more bandwidth
> than we can explain. From what we can tell the replication uses at least
> twice the bandwidth it should.
>
> We have four producer nodes and three broker nodes. We have enabled 3x
> replication, so each node will get a copy of all data in this setup. The
> producers have Snappy compression enabled and send batches of 200 messages.
> The messages are around 1 KiB each. The cluster runs using mostly default
> configuration, and the Kafka version is 0.8.1.1.
>
> When we run iftop on the broker nodes we see that each Kafka node receives
> around 6-7 Mbit from each producer node (or around 25-30 Mbit in total),
> but then sends around 50 Mbit to each other Kafka node (or 100 Mbit in
> total). This is twice what we expected to see, and it seems to saturate the
> bandwidth on our m1.xlarge machines. In other words, we expected the
> incoming 25 Mbit to be amplified to 50 Mbit, not 100.
>
> One thing that could explain it, and that we don't really know how to
> verify, is that the inter-node communication is not compressed. We aren't
> sure about what compression ratio we get on the incoming data, but 50%
> sounds reasonable. Could this explain what we're seeing? Is there a
> configuration property to enable compression on the replication traffic
> that we've missed?
>
> yours
> Theo
>
--
-- Guozhang