You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Christopher Stelly <cd...@gmail.com> on 2016/10/09 16:28:17 UTC
Tuning for high RAM and 10GBe
Hello,
The last thread available regarding 10GBe is about 2 years old, with no
obvious recommendations on tuning.
Is there a more complex tuning guide than the example production config
available on Kafka's main site? Anything other than the list of possible
configs?
I currently have access to a rather substantial academic cluster to test
on, including multiple machines with the following hardware:
10GBe NICs
250GB RAM each
SSDs on each
(also, optional access to single NVMe)
Using Golang's Kafka interface, I can only seem to get about 80MB/s on the
producer pushing to logs on the localhost, using no replication and reading
from/logging to SSD. If it helps, I can post my configs. I've tried
fiddling with a bunch of broker configs as well as producer configs,
raising the memory limits, max message size, io&network threads etc.
Since the last post from 2014 indicates that there is no public
benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on
this hardware if we can get it tuned up properly.
What kind of broker/producer/consumer settings would you recommend?
Thanks!
- chris
Re: Tuning for high RAM and 10GBe
Posted by Eno Thereska <en...@gmail.com>.
Sounds good that you got up to 500MB/s. At that point I suspect you reach a sort of steady state where the cache is continuously flushing to the SSDs, so you are effectively bottlenecked by the SSD. I believe this is as expected (the bottleneck resource will dominate the end to end throughput even if you have a memory buffer to temporarily hold produce records).
If you want to be completely in-memory, that's ok for experimental purposes (e.g, using a RAM-based file system) but you'll then have to start worrying about failures if you are in production. However, if you are interested in just exploring perf for now, using an in-memory file system will be fine. You could even try having 3-way replication, all in memory (with 3 brokers). Again, note of caution: a correlated power failure can lose all your replicated in-memory data, so in production even if you replicate in memory you'd still want to write somewhere persistent. Unless your memory is battery-backed, but that's a longer story.
Cheers
Eno
> On 11 Oct 2016, at 03:49, Christopher Stelly <cd...@gmail.com> wrote:
>
> With that link I came across the producer-perf-test tool, quite useful as
> it gets rid of the Go (Sarama) variable. Since it can quickly tweak
> settings, it's extremely useful.
>
> As you suggested Eno, I attempted to copy the LinkedIn settings. With 100
> byte records, I get up to about 600,000 records/second. Still not quite
> what they were able to do with cheap hardware.
>
> As far as throughput, I tend to max out at around 200MB/s on a single
> producer (record size 10000, no acks, linger 100, batch size 1000000 and
> with compression), along with a generous HEAP_OPT env setting of 50G.
>
> If I do the above settings with 4 producers, things start to slow down. It
> seems to add up to around 500MB/s, which is about what the SSD can write
> at.
>
> Could this number be improved if I let the memory take care of this instead
> of flushing to disk? I understand that Kafka likes to flush often, but even
> relaxing broker's flush settings I can't seem to make an impact on this
> 500MB/s number (w/ 4 producers). After all, we have a lot of RAM to play
> with. The hackish solution would be to make a tempfs mount and store
> kafka-logs there, but that seems like the wrong approach. Any thoughts? Do
> you think flushing is my hold up at this point?
>
> Again, thanks!
>
> On Mon, Oct 10, 2016 at 12:45 PM, Christopher Stelly <cd...@gmail.com>
> wrote:
>
>> Sure, good ideas. I'll try multiple producers, localhost and LAN, to see
>> if any difference
>>
>> Yep, Gwen, the Sarama client. Anything to worry about there outside of
>> setting the producer configs (which would you set?) and number of buffered
>> channels? (currently, buffered channels up to 10k).
>>
>> Thanks!
>>
>> On Mon, Oct 10, 2016 at 12:04 PM, Gwen Shapira <gw...@confluent.io> wrote:
>>
>>> Out of curiosity - what is "Golang's Kafka interface"? Are you
>>> referring to Sarama client?
>>>
>>> On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cd...@gmail.com>
>>> wrote:
>>>> Hello,
>>>>
>>>> The last thread available regarding 10GBe is about 2 years old, with no
>>>> obvious recommendations on tuning.
>>>>
>>>> Is there a more complex tuning guide than the example production config
>>>> available on Kafka's main site? Anything other than the list of possible
>>>> configs?
>>>>
>>>> I currently have access to a rather substantial academic cluster to test
>>>> on, including multiple machines with the following hardware:
>>>>
>>>> 10GBe NICs
>>>> 250GB RAM each
>>>> SSDs on each
>>>> (also, optional access to single NVMe)
>>>>
>>>> Using Golang's Kafka interface, I can only seem to get about 80MB/s on
>>> the
>>>> producer pushing to logs on the localhost, using no replication and
>>> reading
>>>> from/logging to SSD. If it helps, I can post my configs. I've tried
>>>> fiddling with a bunch of broker configs as well as producer configs,
>>>> raising the memory limits, max message size, io&network threads etc.
>>>>
>>>> Since the last post from 2014 indicates that there is no public
>>>> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results
>>> on
>>>> this hardware if we can get it tuned up properly.
>>>>
>>>> What kind of broker/producer/consumer settings would you recommend?
>>>>
>>>> Thanks!
>>>> - chris
>>>
>>>
>>>
>>> --
>>> Gwen Shapira
>>> Product Manager | Confluent
>>> 650.450.2760 | @gwenshap
>>> Follow us: Twitter | blog
>>>
>>
>>
Re: Tuning for high RAM and 10GBe
Posted by Christopher Stelly <cd...@gmail.com>.
With that link I came across the producer-perf-test tool, quite useful as
it gets rid of the Go (Sarama) variable. Since it can quickly tweak
settings, it's extremely useful.
As you suggested Eno, I attempted to copy the LinkedIn settings. With 100
byte records, I get up to about 600,000 records/second. Still not quite
what they were able to do with cheap hardware.
As far as throughput, I tend to max out at around 200MB/s on a single
producer (record size 10000, no acks, linger 100, batch size 1000000 and
with compression), along with a generous HEAP_OPT env setting of 50G.
If I do the above settings with 4 producers, things start to slow down. It
seems to add up to around 500MB/s, which is about what the SSD can write
at.
Could this number be improved if I let the memory take care of this instead
of flushing to disk? I understand that Kafka likes to flush often, but even
relaxing broker's flush settings I can't seem to make an impact on this
500MB/s number (w/ 4 producers). After all, we have a lot of RAM to play
with. The hackish solution would be to make a tempfs mount and store
kafka-logs there, but that seems like the wrong approach. Any thoughts? Do
you think flushing is my hold up at this point?
Again, thanks!
On Mon, Oct 10, 2016 at 12:45 PM, Christopher Stelly <cd...@gmail.com>
wrote:
> Sure, good ideas. I'll try multiple producers, localhost and LAN, to see
> if any difference
>
> Yep, Gwen, the Sarama client. Anything to worry about there outside of
> setting the producer configs (which would you set?) and number of buffered
> channels? (currently, buffered channels up to 10k).
>
> Thanks!
>
> On Mon, Oct 10, 2016 at 12:04 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
>> Out of curiosity - what is "Golang's Kafka interface"? Are you
>> referring to Sarama client?
>>
>> On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cd...@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > The last thread available regarding 10GBe is about 2 years old, with no
>> > obvious recommendations on tuning.
>> >
>> > Is there a more complex tuning guide than the example production config
>> > available on Kafka's main site? Anything other than the list of possible
>> > configs?
>> >
>> > I currently have access to a rather substantial academic cluster to test
>> > on, including multiple machines with the following hardware:
>> >
>> > 10GBe NICs
>> > 250GB RAM each
>> > SSDs on each
>> > (also, optional access to single NVMe)
>> >
>> > Using Golang's Kafka interface, I can only seem to get about 80MB/s on
>> the
>> > producer pushing to logs on the localhost, using no replication and
>> reading
>> > from/logging to SSD. If it helps, I can post my configs. I've tried
>> > fiddling with a bunch of broker configs as well as producer configs,
>> > raising the memory limits, max message size, io&network threads etc.
>> >
>> > Since the last post from 2014 indicates that there is no public
>> > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results
>> on
>> > this hardware if we can get it tuned up properly.
>> >
>> > What kind of broker/producer/consumer settings would you recommend?
>> >
>> > Thanks!
>> > - chris
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>
>
Re: Tuning for high RAM and 10GBe
Posted by Christopher Stelly <cd...@gmail.com>.
Sure, good ideas. I'll try multiple producers, localhost and LAN, to see if
any difference
Yep, Gwen, the Sarama client. Anything to worry about there outside of
setting the producer configs (which would you set?) and number of buffered
channels? (currently, buffered channels up to 10k).
Thanks!
On Mon, Oct 10, 2016 at 12:04 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Out of curiosity - what is "Golang's Kafka interface"? Are you
> referring to Sarama client?
>
> On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cd...@gmail.com>
> wrote:
> > Hello,
> >
> > The last thread available regarding 10GBe is about 2 years old, with no
> > obvious recommendations on tuning.
> >
> > Is there a more complex tuning guide than the example production config
> > available on Kafka's main site? Anything other than the list of possible
> > configs?
> >
> > I currently have access to a rather substantial academic cluster to test
> > on, including multiple machines with the following hardware:
> >
> > 10GBe NICs
> > 250GB RAM each
> > SSDs on each
> > (also, optional access to single NVMe)
> >
> > Using Golang's Kafka interface, I can only seem to get about 80MB/s on
> the
> > producer pushing to logs on the localhost, using no replication and
> reading
> > from/logging to SSD. If it helps, I can post my configs. I've tried
> > fiddling with a bunch of broker configs as well as producer configs,
> > raising the memory limits, max message size, io&network threads etc.
> >
> > Since the last post from 2014 indicates that there is no public
> > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results
> on
> > this hardware if we can get it tuned up properly.
> >
> > What kind of broker/producer/consumer settings would you recommend?
> >
> > Thanks!
> > - chris
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
Re: Tuning for high RAM and 10GBe
Posted by Gwen Shapira <gw...@confluent.io>.
Out of curiosity - what is "Golang's Kafka interface"? Are you
referring to Sarama client?
On Sun, Oct 9, 2016 at 9:28 AM, Christopher Stelly <cd...@gmail.com> wrote:
> Hello,
>
> The last thread available regarding 10GBe is about 2 years old, with no
> obvious recommendations on tuning.
>
> Is there a more complex tuning guide than the example production config
> available on Kafka's main site? Anything other than the list of possible
> configs?
>
> I currently have access to a rather substantial academic cluster to test
> on, including multiple machines with the following hardware:
>
> 10GBe NICs
> 250GB RAM each
> SSDs on each
> (also, optional access to single NVMe)
>
> Using Golang's Kafka interface, I can only seem to get about 80MB/s on the
> producer pushing to logs on the localhost, using no replication and reading
> from/logging to SSD. If it helps, I can post my configs. I've tried
> fiddling with a bunch of broker configs as well as producer configs,
> raising the memory limits, max message size, io&network threads etc.
>
> Since the last post from 2014 indicates that there is no public
> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on
> this hardware if we can get it tuned up properly.
>
> What kind of broker/producer/consumer settings would you recommend?
>
> Thanks!
> - chris
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: Tuning for high RAM and 10GBe
Posted by Eno Thereska <en...@gmail.com>.
Hi Chris,
I think the first step would be to set up the system so we can easily identify the bottlenecks. With your setup I'm currently worried about 2 things:
1. the system is not being driven hard enough. In particular 1 producer might not be enough. I'd recommend running 3 producer processes.
2. CPU interference between producers and Kafka itself, since they are on the same box. Given that you have 10Gbps you should probably set up the producer on a different machine to Kafka (so a 3 machine setup: producer, zk, 1 kafka broker).
With these two in place, the next thing to look at would be perf metrics to see if CPU, network and storage is the bottleneck. Would you have access to those metrics while the test is running.
Also, for calibration, I'd run a test with small (100 byte) values. That way we can compare to https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines <https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines> for example.
Does this help?
Thanks
Eno
> On 9 Oct 2016, at 18:19, Christopher Stelly <cd...@gmail.com> wrote:
>
> Hey Eno, thanks
>
> Yes, localhost would mean testing SSD, maybe moving to a remote producer in
> the future over 10GBe but baby steps first.
>
> My record size is pretty flexible - I've tested from 4k to 16M and the
> results are +- 10MB/s.
>
> SSD is a samsung 850, claiming 100k IOPS and >500MB/s read/write.
>
> Again, thanks!
>
> On Sun, Oct 9, 2016 at 12:55 PM, Eno Thereska <en...@gmail.com>
> wrote:
>
>> Hi Chris,
>>
>> A couple of things: looks like you're primarily testing the SSD if you're
>> running on the localhost, right? The 10GBe shouldn't matter in this case.
>>
>> The performance will depend a lot on the record sizes you are using. To
>> get a ballpark number if would help to know the SSD type, would you be
>> willing to share it? We can take it from there.
>>
>> Thanks
>> Eno
>>
>>> On 9 Oct 2016, at 17:28, Christopher Stelly <cd...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> The last thread available regarding 10GBe is about 2 years old, with no
>>> obvious recommendations on tuning.
>>>
>>> Is there a more complex tuning guide than the example production config
>>> available on Kafka's main site? Anything other than the list of possible
>>> configs?
>>>
>>> I currently have access to a rather substantial academic cluster to test
>>> on, including multiple machines with the following hardware:
>>>
>>> 10GBe NICs
>>> 250GB RAM each
>>> SSDs on each
>>> (also, optional access to single NVMe)
>>>
>>> Using Golang's Kafka interface, I can only seem to get about 80MB/s on
>> the
>>> producer pushing to logs on the localhost, using no replication and
>> reading
>>> from/logging to SSD. If it helps, I can post my configs. I've tried
>>> fiddling with a bunch of broker configs as well as producer configs,
>>> raising the memory limits, max message size, io&network threads etc.
>>>
>>> Since the last post from 2014 indicates that there is no public
>>> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results
>> on
>>> this hardware if we can get it tuned up properly.
>>>
>>> What kind of broker/producer/consumer settings would you recommend?
>>>
>>> Thanks!
>>> - chris
>>
>>
Re: Tuning for high RAM and 10GBe
Posted by Christopher Stelly <cd...@gmail.com>.
Hey Eno, thanks
Yes, localhost would mean testing SSD, maybe moving to a remote producer in
the future over 10GBe but baby steps first.
My record size is pretty flexible - I've tested from 4k to 16M and the
results are +- 10MB/s.
SSD is a samsung 850, claiming 100k IOPS and >500MB/s read/write.
Again, thanks!
On Sun, Oct 9, 2016 at 12:55 PM, Eno Thereska <en...@gmail.com>
wrote:
> Hi Chris,
>
> A couple of things: looks like you're primarily testing the SSD if you're
> running on the localhost, right? The 10GBe shouldn't matter in this case.
>
> The performance will depend a lot on the record sizes you are using. To
> get a ballpark number if would help to know the SSD type, would you be
> willing to share it? We can take it from there.
>
> Thanks
> Eno
>
> > On 9 Oct 2016, at 17:28, Christopher Stelly <cd...@gmail.com> wrote:
> >
> > Hello,
> >
> > The last thread available regarding 10GBe is about 2 years old, with no
> > obvious recommendations on tuning.
> >
> > Is there a more complex tuning guide than the example production config
> > available on Kafka's main site? Anything other than the list of possible
> > configs?
> >
> > I currently have access to a rather substantial academic cluster to test
> > on, including multiple machines with the following hardware:
> >
> > 10GBe NICs
> > 250GB RAM each
> > SSDs on each
> > (also, optional access to single NVMe)
> >
> > Using Golang's Kafka interface, I can only seem to get about 80MB/s on
> the
> > producer pushing to logs on the localhost, using no replication and
> reading
> > from/logging to SSD. If it helps, I can post my configs. I've tried
> > fiddling with a bunch of broker configs as well as producer configs,
> > raising the memory limits, max message size, io&network threads etc.
> >
> > Since the last post from 2014 indicates that there is no public
> > benchmarking for 10GBe, I'd be happy to run benchmarks /publish results
> on
> > this hardware if we can get it tuned up properly.
> >
> > What kind of broker/producer/consumer settings would you recommend?
> >
> > Thanks!
> > - chris
>
>
Re: Tuning for high RAM and 10GBe
Posted by Eno Thereska <en...@gmail.com>.
Hi Chris,
A couple of things: looks like you're primarily testing the SSD if you're running on the localhost, right? The 10GBe shouldn't matter in this case.
The performance will depend a lot on the record sizes you are using. To get a ballpark number if would help to know the SSD type, would you be willing to share it? We can take it from there.
Thanks
Eno
> On 9 Oct 2016, at 17:28, Christopher Stelly <cd...@gmail.com> wrote:
>
> Hello,
>
> The last thread available regarding 10GBe is about 2 years old, with no
> obvious recommendations on tuning.
>
> Is there a more complex tuning guide than the example production config
> available on Kafka's main site? Anything other than the list of possible
> configs?
>
> I currently have access to a rather substantial academic cluster to test
> on, including multiple machines with the following hardware:
>
> 10GBe NICs
> 250GB RAM each
> SSDs on each
> (also, optional access to single NVMe)
>
> Using Golang's Kafka interface, I can only seem to get about 80MB/s on the
> producer pushing to logs on the localhost, using no replication and reading
> from/logging to SSD. If it helps, I can post my configs. I've tried
> fiddling with a bunch of broker configs as well as producer configs,
> raising the memory limits, max message size, io&network threads etc.
>
> Since the last post from 2014 indicates that there is no public
> benchmarking for 10GBe, I'd be happy to run benchmarks /publish results on
> this hardware if we can get it tuned up properly.
>
> What kind of broker/producer/consumer settings would you recommend?
>
> Thanks!
> - chris