You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Gioacchino Vino <gi...@gmail.com> on 2019/01/08 17:45:25 UTC
Configuration guidelines for a specific use-case
Hi expert,
I would ask you some guidelines, web-pages or comments regarding my
use-case.
*Requirements*:
- 2000+ producers
- input rate 600k messages/s
- consumers must write in 3 different databases (so i assume 3 consumer
groups) at 600k messages/s overall (200k messages/s/database)
- latency < 500ms between producers and databases
- good availability
- Possibility to process messages before to send them to the databases
(Kafka stream? Of course in HA. Docker? Marathon?)
- it's tolerate missing data ( 0.5% max ) (disk writing is not strictly
required), latency has higher priority
- record size: 100-1000
*Resources*:
brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)
producers -> brokers -> consumers ( Bandwidth: 1 Gbps )
*My* *configuration*:
3 brokers
6 partition (without replication in order to minimize latency)
ack = 0 (missing data is tolerate)
batch.size = 1024 (with 8196 the throughput is max)
producers -> compression.type=none
I did test using kafka-producer-perf-test.sh and
kafka-consumer-perf-test.sh and i have a good throughput (500-600k
messages/s using 3 producers and 3 consumers) but i would improve
latency (0.3-2 sec) or features I'm not still considering.
I thank you in advance.
Cheers,
Gioacchino
Re: Configuration guidelines for a specific use-case
Posted by Ryanne Dolan <ry...@gmail.com>.
Given your high throughput on the consumer side, you might consider adding
more partitions than 6, so you can scale up beyond 6 consumers per group if
need be.
Ryanne
On Wed, Jan 9, 2019 at 10:37 AM Gioacchino Vino <gi...@gmail.com>
wrote:
> Hi Ryanne,
>
>
> I just forgot to insert the "linger.ms=0" configuration.
>
> I got this result:
>
>
> 50000000 records sent, 706793.701055 records/sec (67.41 MB/sec), 7.29 ms
> avg latency, 1245.00 ms max latency, 0 ms 50th, 3 ms 95th, 197 ms 99th,
> 913 ms 99.9th.
>
>
> it's pretty good but I would like to improve it just a bit.
>
> Do you think using 6 partitions in a 3 broker cluster is a good choice?
>
>
> Gioacchino
>
>
> On 08/01/2019 18:52, Ryanne Dolan wrote:
> > Latency sounds high to me, maybe your JVMs are GC'ing a lot?
> >
> > Ryanne
> >
> > On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinovino@gmail.com
> > wrote:
> >
> >> Hi expert,
> >>
> >>
> >> I would ask you some guidelines, web-pages or comments regarding my
> >> use-case.
> >>
> >>
> >> *Requirements*:
> >>
> >> - 2000+ producers
> >>
> >> - input rate 600k messages/s
> >>
> >> - consumers must write in 3 different databases (so i assume 3 consumer
> >> groups) at 600k messages/s overall (200k messages/s/database)
> >>
> >> - latency < 500ms between producers and databases
> >>
> >> - good availability
> >>
> >> - Possibility to process messages before to send them to the databases
> >> (Kafka stream? Of course in HA. Docker? Marathon?)
> >>
> >> - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly
> >> required), latency has higher priority
> >>
> >> - record size: 100-1000
> >>
> >>
> >> *Resources*:
> >>
> >> brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)
> >>
> >> producers -> brokers -> consumers ( Bandwidth: 1 Gbps )
> >>
> >>
> >> *My* *configuration*:
> >>
> >> 3 brokers
> >>
> >> 6 partition (without replication in order to minimize latency)
> >>
> >> ack = 0 (missing data is tolerate)
> >>
> >> batch.size = 1024 (with 8196 the throughput is max)
> >>
> >> producers -> compression.type=none
> >>
> >>
> >>
> >> I did test using kafka-producer-perf-test.sh and
> >> kafka-consumer-perf-test.sh and i have a good throughput (500-600k
> >> messages/s using 3 producers and 3 consumers) but i would improve
> >> latency (0.3-2 sec) or features I'm not still considering.
> >>
> >>
> >> I thank you in advance.
> >>
> >> Cheers,
> >>
> >>
> >> Gioacchino
> >>
> >>
>
Re: Configuration guidelines for a specific use-case
Posted by Gioacchino Vino <gi...@gmail.com>.
Hi Ryanne,
I just forgot to insert the "linger.ms=0" configuration.
I got this result:
50000000 records sent, 706793.701055 records/sec (67.41 MB/sec), 7.29 ms
avg latency, 1245.00 ms max latency, 0 ms 50th, 3 ms 95th, 197 ms 99th,
913 ms 99.9th.
it's pretty good but I would like to improve it just a bit.
Do you think using 6 partitions in a 3 broker cluster is a good choice?
Gioacchino
On 08/01/2019 18:52, Ryanne Dolan wrote:
> Latency sounds high to me, maybe your JVMs are GC'ing a lot?
>
> Ryanne
>
> On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinovino@gmail.com
> wrote:
>
>> Hi expert,
>>
>>
>> I would ask you some guidelines, web-pages or comments regarding my
>> use-case.
>>
>>
>> *Requirements*:
>>
>> - 2000+ producers
>>
>> - input rate 600k messages/s
>>
>> - consumers must write in 3 different databases (so i assume 3 consumer
>> groups) at 600k messages/s overall (200k messages/s/database)
>>
>> - latency < 500ms between producers and databases
>>
>> - good availability
>>
>> - Possibility to process messages before to send them to the databases
>> (Kafka stream? Of course in HA. Docker? Marathon?)
>>
>> - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly
>> required), latency has higher priority
>>
>> - record size: 100-1000
>>
>>
>> *Resources*:
>>
>> brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)
>>
>> producers -> brokers -> consumers ( Bandwidth: 1 Gbps )
>>
>>
>> *My* *configuration*:
>>
>> 3 brokers
>>
>> 6 partition (without replication in order to minimize latency)
>>
>> ack = 0 (missing data is tolerate)
>>
>> batch.size = 1024 (with 8196 the throughput is max)
>>
>> producers -> compression.type=none
>>
>>
>>
>> I did test using kafka-producer-perf-test.sh and
>> kafka-consumer-perf-test.sh and i have a good throughput (500-600k
>> messages/s using 3 producers and 3 consumers) but i would improve
>> latency (0.3-2 sec) or features I'm not still considering.
>>
>>
>> I thank you in advance.
>>
>> Cheers,
>>
>>
>> Gioacchino
>>
>>
Re: Configuration guidelines for a specific use-case
Posted by Ryanne Dolan <ry...@gmail.com>.
Latency sounds high to me, maybe your JVMs are GC'ing a lot?
Ryanne
On Tue, Jan 8, 2019, 11:45 AM Gioacchino Vino <gioacchinovino@gmail.com
wrote:
> Hi expert,
>
>
> I would ask you some guidelines, web-pages or comments regarding my
> use-case.
>
>
> *Requirements*:
>
> - 2000+ producers
>
> - input rate 600k messages/s
>
> - consumers must write in 3 different databases (so i assume 3 consumer
> groups) at 600k messages/s overall (200k messages/s/database)
>
> - latency < 500ms between producers and databases
>
> - good availability
>
> - Possibility to process messages before to send them to the databases
> (Kafka stream? Of course in HA. Docker? Marathon?)
>
> - it's tolerate missing data ( 0.5% max ) (disk writing is not strictly
> required), latency has higher priority
>
> - record size: 100-1000
>
>
> *Resources*:
>
> brokers ( Bandwidth: 25 Gbps, 32Cpus, 1 disk (I/O 99.0 MB/s)
>
> producers -> brokers -> consumers ( Bandwidth: 1 Gbps )
>
>
> *My* *configuration*:
>
> 3 brokers
>
> 6 partition (without replication in order to minimize latency)
>
> ack = 0 (missing data is tolerate)
>
> batch.size = 1024 (with 8196 the throughput is max)
>
> producers -> compression.type=none
>
>
>
> I did test using kafka-producer-perf-test.sh and
> kafka-consumer-perf-test.sh and i have a good throughput (500-600k
> messages/s using 3 producers and 3 consumers) but i would improve
> latency (0.3-2 sec) or features I'm not still considering.
>
>
> I thank you in advance.
>
> Cheers,
>
>
> Gioacchino
>
>