You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Shlomi Hazan <sh...@viber.com> on 2014/12/29 12:26:35 UTC
latency - how to reduce?
Hi,
I am using 0.8.1.1, and I have hundreds of msec latency at best and even
seconds at worst.
I have this latency both on production, (with peak load of 30K msg/sec,
replication = 2 across 5 brokers, acks = 1),
and on the local windows machine using just one process for each of
producer, zookeeper, kafka, consumer.
Also tried batch.num.messages=1 and producer.type=sync on the local machine
but saw no improvement.
How can I push latency down to several millis, at least when running local?
Thanks,
Shlomi
Re: latency - how to reduce?
Posted by Jun Rao <ju...@confluent.io>.
You probably want to test at least the sequential write latency (with
flush) on your file system.
Thanks,
Jun
On Mon, Jan 12, 2015 at 6:08 AM, Shlomi Hazan <sh...@viber.com> wrote:
> Thank you for the guidance, yet I want to know I am doing the test Jun had
> in mind
>
> On Thu, Jan 8, 2015 at 5:36 PM, Jayesh Thakrar <j_thakrar@yahoo.com.invalid
> >
> wrote:
>
> > I do see the Windows based scripts in the tar file - but haven't them
> > though.You should find them under bin/windows.
> > Also you can always use other Windows stress testing tools/suites to
> check
> > your local I/O performance..
> > From: Shlomi Hazan <sh...@viber.com>
> > To: users@kafka.apache.org; Jayesh Thakrar <j_...@yahoo.com>
> > Sent: Thursday, January 8, 2015 6:25 AM
> > Subject: Re: latency - how to reduce?
> >
> > I would like to test locally first as it is easier than setting up a test
> > cluster to model the production, yet the script kafka-producer-perf-test
> is
> > not available for windows.
> > Jun, what kind of "basic I/O testing on the local FS" did you have in
> mind?
> >
> > Thanks,
> > Shlomi
> >
> >
> >
> >
> > On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar
> <j_thakrar@yahoo.com.invalid
> > >
> > wrote:
> >
> > > Have you tried using the built-in stress test scripts?
> > > bin/kafka-producer-perf-test.sh
> > > bin/kafka-consumer-perf-test.sh
> > >
> > > Here's how I stress tested them -
> > > nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> > > ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> > > --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
> > >
> > > nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> > > ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> > > 1>kafka-consumer-perf-test.sh.log 2>&1 &
> > >
> > > And I used screen scrapping of the jmx ui screens to push metrics into
> > > TSDB to get the following.The rate below is per second - so I could
> push
> > > the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> > > little utilization (<30% utilization).
> > >
> > >
> > > From: Shlomi Hazan <sh...@viber.com>
> > > To: users@kafka.apache.org
> > > Sent: Tuesday, January 6, 2015 1:06 AM
> > > Subject: Re: latency - how to reduce?
> > >
> > > Will do. What did you have in mind? just write a big file to disk and
> > > measure the time it took to write? maybe also read back? using specific
> > > API's?
> > > Apart from the local Win machine case, are you aware of any issues with
> > > Amazon EC2 instances that may be causing that same latency in
> production?
> > > Thanks,
> > > Shlomi
> > >
> > >
> > >
> > > On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Not setting "log.flush.interval.messages" is good since the default
> > gives
> > > > the best latency. Could you do some basic I/O testing on the local FS
> > in
> > > > your windows machine to make sure the I/O latency is ok?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com>
> wrote:
> > > >
> > > > > Happy new year!
> > > > > I did not set "log.flush.interval.messages".
> > > > > I also could not find a default value in the docs.
> > > > > Could you explain about that?
> > > > > Thanks,
> > > > > Shlomi
> > > > >
> > > > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> > > > >
> > > > > > What's your setting of log.flush.interval.messages on the broker?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best
> > and
> > > > > even
> > > > > > > seconds at worst.
> > > > > > > I have this latency both on production, (with peak load of 30K
> > > > msg/sec,
> > > > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > > > and on the local windows machine using just one process for
> each
> > of
> > > > > > > producer, zookeeper, kafka, consumer.
> > > > > > > Also tried batch.num.messages=1 and producer.type=sync on the
> > local
> > > > > > machine
> > > > > > > but saw no improvement.
> > > > > > > How can I push latency down to several millis, at least when
> > > running
> > > > > > local?
> > > > > > > Thanks,
> > > > > > > Shlomi
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
Re: latency - how to reduce?
Posted by Shlomi Hazan <sh...@viber.com>.
Thank you for the guidance, yet I want to know I am doing the test Jun had
in mind
On Thu, Jan 8, 2015 at 5:36 PM, Jayesh Thakrar <j_...@yahoo.com.invalid>
wrote:
> I do see the Windows based scripts in the tar file - but haven't them
> though.You should find them under bin/windows.
> Also you can always use other Windows stress testing tools/suites to check
> your local I/O performance..
> From: Shlomi Hazan <sh...@viber.com>
> To: users@kafka.apache.org; Jayesh Thakrar <j_...@yahoo.com>
> Sent: Thursday, January 8, 2015 6:25 AM
> Subject: Re: latency - how to reduce?
>
> I would like to test locally first as it is easier than setting up a test
> cluster to model the production, yet the script kafka-producer-perf-test is
> not available for windows.
> Jun, what kind of "basic I/O testing on the local FS" did you have in mind?
>
> Thanks,
> Shlomi
>
>
>
>
> On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar <j_thakrar@yahoo.com.invalid
> >
> wrote:
>
> > Have you tried using the built-in stress test scripts?
> > bin/kafka-producer-perf-test.sh
> > bin/kafka-consumer-perf-test.sh
> >
> > Here's how I stress tested them -
> > nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> > ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> > --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
> >
> > nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> > ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> > 1>kafka-consumer-perf-test.sh.log 2>&1 &
> >
> > And I used screen scrapping of the jmx ui screens to push metrics into
> > TSDB to get the following.The rate below is per second - so I could push
> > the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> > little utilization (<30% utilization).
> >
> >
> > From: Shlomi Hazan <sh...@viber.com>
> > To: users@kafka.apache.org
> > Sent: Tuesday, January 6, 2015 1:06 AM
> > Subject: Re: latency - how to reduce?
> >
> > Will do. What did you have in mind? just write a big file to disk and
> > measure the time it took to write? maybe also read back? using specific
> > API's?
> > Apart from the local Win machine case, are you aware of any issues with
> > Amazon EC2 instances that may be causing that same latency in production?
> > Thanks,
> > Shlomi
> >
> >
> >
> > On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Not setting "log.flush.interval.messages" is good since the default
> gives
> > > the best latency. Could you do some basic I/O testing on the local FS
> in
> > > your windows machine to make sure the I/O latency is ok?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
> > >
> > > > Happy new year!
> > > > I did not set "log.flush.interval.messages".
> > > > I also could not find a default value in the docs.
> > > > Could you explain about that?
> > > > Thanks,
> > > > Shlomi
> > > >
> > > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> > > >
> > > > > What's your setting of log.flush.interval.messages on the broker?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
> > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best
> and
> > > > even
> > > > > > seconds at worst.
> > > > > > I have this latency both on production, (with peak load of 30K
> > > msg/sec,
> > > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > > and on the local windows machine using just one process for each
> of
> > > > > > producer, zookeeper, kafka, consumer.
> > > > > > Also tried batch.num.messages=1 and producer.type=sync on the
> local
> > > > > machine
> > > > > > but saw no improvement.
> > > > > > How can I push latency down to several millis, at least when
> > running
> > > > > local?
> > > > > > Thanks,
> > > > > > Shlomi
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
>
>
>
Re: latency - how to reduce?
Posted by Jayesh Thakrar <j_...@yahoo.com.INVALID>.
I do see the Windows based scripts in the tar file - but haven't them though.You should find them under bin/windows.
Also you can always use other Windows stress testing tools/suites to check your local I/O performance..
From: Shlomi Hazan <sh...@viber.com>
To: users@kafka.apache.org; Jayesh Thakrar <j_...@yahoo.com>
Sent: Thursday, January 8, 2015 6:25 AM
Subject: Re: latency - how to reduce?
I would like to test locally first as it is easier than setting up a test
cluster to model the production, yet the script kafka-producer-perf-test is
not available for windows.
Jun, what kind of "basic I/O testing on the local FS" did you have in mind?
Thanks,
Shlomi
On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar <j_...@yahoo.com.invalid>
wrote:
> Have you tried using the built-in stress test scripts?
> bin/kafka-producer-perf-test.sh
> bin/kafka-consumer-perf-test.sh
>
> Here's how I stress tested them -
> nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
>
> nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> 1>kafka-consumer-perf-test.sh.log 2>&1 &
>
> And I used screen scrapping of the jmx ui screens to push metrics into
> TSDB to get the following.The rate below is per second - so I could push
> the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> little utilization (<30% utilization).
>
>
> From: Shlomi Hazan <sh...@viber.com>
> To: users@kafka.apache.org
> Sent: Tuesday, January 6, 2015 1:06 AM
> Subject: Re: latency - how to reduce?
>
> Will do. What did you have in mind? just write a big file to disk and
> measure the time it took to write? maybe also read back? using specific
> API's?
> Apart from the local Win machine case, are you aware of any issues with
> Amazon EC2 instances that may be causing that same latency in production?
> Thanks,
> Shlomi
>
>
>
> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Not setting "log.flush.interval.messages" is good since the default gives
> > the best latency. Could you do some basic I/O testing on the local FS in
> > your windows machine to make sure the I/O latency is ok?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
> >
> > > Happy new year!
> > > I did not set "log.flush.interval.messages".
> > > I also could not find a default value in the docs.
> > > Could you explain about that?
> > > Thanks,
> > > Shlomi
> > >
> > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > What's your setting of log.flush.interval.messages on the broker?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> > > even
> > > > > seconds at worst.
> > > > > I have this latency both on production, (with peak load of 30K
> > msg/sec,
> > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > and on the local windows machine using just one process for each of
> > > > > producer, zookeeper, kafka, consumer.
> > > > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > > > machine
> > > > > but saw no improvement.
> > > > > How can I push latency down to several millis, at least when
> running
> > > > local?
> > > > > Thanks,
> > > > > Shlomi
> > > > >
> > > >
> > >
> >
>
>
>
>
Re: latency - how to reduce?
Posted by Shlomi Hazan <sh...@viber.com>.
I would like to test locally first as it is easier than setting up a test
cluster to model the production, yet the script kafka-producer-perf-test is
not available for windows.
Jun, what kind of "basic I/O testing on the local FS" did you have in mind?
Thanks,
Shlomi
On Tue, Jan 6, 2015 at 5:40 PM, Jayesh Thakrar <j_...@yahoo.com.invalid>
wrote:
> Have you tried using the built-in stress test scripts?
> bin/kafka-producer-perf-test.sh
> bin/kafka-consumer-perf-test.sh
>
> Here's how I stress tested them -
> nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list
> ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16
> --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
>
> nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper
> ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16
> 1>kafka-consumer-perf-test.sh.log 2>&1 &
>
> And I used screen scrapping of the jmx ui screens to push metrics into
> TSDB to get the following.The rate below is per second - so I could push
> the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very
> little utilization (<30% utilization).
>
>
> From: Shlomi Hazan <sh...@viber.com>
> To: users@kafka.apache.org
> Sent: Tuesday, January 6, 2015 1:06 AM
> Subject: Re: latency - how to reduce?
>
> Will do. What did you have in mind? just write a big file to disk and
> measure the time it took to write? maybe also read back? using specific
> API's?
> Apart from the local Win machine case, are you aware of any issues with
> Amazon EC2 instances that may be causing that same latency in production?
> Thanks,
> Shlomi
>
>
>
> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Not setting "log.flush.interval.messages" is good since the default gives
> > the best latency. Could you do some basic I/O testing on the local FS in
> > your windows machine to make sure the I/O latency is ok?
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
> >
> > > Happy new year!
> > > I did not set "log.flush.interval.messages".
> > > I also could not find a default value in the docs.
> > > Could you explain about that?
> > > Thanks,
> > > Shlomi
> > >
> > > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > What's your setting of log.flush.interval.messages on the broker?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
> > wrote:
> > > >
> > > > > Hi,
> > > > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> > > even
> > > > > seconds at worst.
> > > > > I have this latency both on production, (with peak load of 30K
> > msg/sec,
> > > > > replication = 2 across 5 brokers, acks = 1),
> > > > > and on the local windows machine using just one process for each of
> > > > > producer, zookeeper, kafka, consumer.
> > > > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > > > machine
> > > > > but saw no improvement.
> > > > > How can I push latency down to several millis, at least when
> running
> > > > local?
> > > > > Thanks,
> > > > > Shlomi
> > > > >
> > > >
> > >
> >
>
>
>
>
Re: latency - how to reduce?
Posted by Jayesh Thakrar <j_...@yahoo.com.INVALID>.
Have you tried using the built-in stress test scripts?
bin/kafka-producer-perf-test.sh
bin/kafka-consumer-perf-test.sh
Here's how I stress tested them -
nohup ${KAFKA_HOME}/bin/kafka-producer-perf-test.sh --broker-list ${KAFKA_SERVERS} --topic ${TOPIC_NAME} --new-producer --threads 16 --messages 100000000 1>kafka-producer-perf-test.sh.log 2>&1 &
nohup ${KAFKA_HOME}/bin/kafka-consumer-perf-test.sh --zookeeper ${ZOOKEEPER_QUORUM} --topic ${TOPIC_NAME} --threads 16 1>kafka-consumer-perf-test.sh.log 2>&1 &
And I used screen scrapping of the jmx ui screens to push metrics into TSDB to get the following.The rate below is per second - so I could push the Kafka cluster to 140k+ messages/sec on a 4-node cluster with very little utilization (<30% utilization).
From: Shlomi Hazan <sh...@viber.com>
To: users@kafka.apache.org
Sent: Tuesday, January 6, 2015 1:06 AM
Subject: Re: latency - how to reduce?
Will do. What did you have in mind? just write a big file to disk and
measure the time it took to write? maybe also read back? using specific
API's?
Apart from the local Win machine case, are you aware of any issues with
Amazon EC2 instances that may be causing that same latency in production?
Thanks,
Shlomi
On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
> Not setting "log.flush.interval.messages" is good since the default gives
> the best latency. Could you do some basic I/O testing on the local FS in
> your windows machine to make sure the I/O latency is ok?
>
> Thanks,
>
> Jun
>
> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
>
> > Happy new year!
> > I did not set "log.flush.interval.messages".
> > I also could not find a default value in the docs.
> > Could you explain about that?
> > Thanks,
> > Shlomi
> >
> > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > What's your setting of log.flush.interval.messages on the broker?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
> wrote:
> > >
> > > > Hi,
> > > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> > even
> > > > seconds at worst.
> > > > I have this latency both on production, (with peak load of 30K
> msg/sec,
> > > > replication = 2 across 5 brokers, acks = 1),
> > > > and on the local windows machine using just one process for each of
> > > > producer, zookeeper, kafka, consumer.
> > > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > > machine
> > > > but saw no improvement.
> > > > How can I push latency down to several millis, at least when running
> > > local?
> > > > Thanks,
> > > > Shlomi
> > > >
> > >
> >
>
Re: latency - how to reduce?
Posted by Shannon Lloyd <sh...@gmail.com>.
This is a good point, even though you mentioned that you also have latency
issues locally. I just migrated a 3-node test cluster from m3.large
instances to c4.xlarge instances (3-node ZK migrated from m3.medium to
c4.large) in an EC2 placement group (better network IO and more consistent
latencies), and both throughput and latencies improved dramatically. We're
deliberately using ack all with replication factor of 2 and only 1
partition, and even with this we comfortably hit 10K messages/sec with sub
ms latency, and 100K messages per sec with latencies in the 4-5ms range. On
the m3.large cluster we maxed out at a couple of thousand messages/sec. And
we haven't tuned any settings for performance (quite the opposite!), other
than to write all logs to dedicated EBS volumes.
What instance types are you using, both for ZK and for Kafka? The lower
end, general purpose instance types really aren't that spectacular,
especially with regard to networking. And look into AWS placement groups.
Cheers,
Shannon
On 19 Jan 2015 16:52, "Andrew Ehrlich" <an...@aehrlich.com> wrote:
> What instance types are you using in EC2? Are the drives EBS?
>
> On 1/5/15 11:06 PM, Shlomi Hazan wrote:
>
>> Will do. What did you have in mind? just write a big file to disk and
>> measure the time it took to write? maybe also read back? using specific
>> API's?
>> Apart from the local Win machine case, are you aware of any issues with
>> Amazon EC2 instances that may be causing that same latency in production?
>> Thanks,
>> Shlomi
>>
>> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> Not setting "log.flush.interval.messages" is good since the default gives
>>> the best latency. Could you do some basic I/O testing on the local FS in
>>> your windows machine to make sure the I/O latency is ok?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
>>>
>>> Happy new year!
>>>> I did not set "log.flush.interval.messages".
>>>> I also could not find a default value in the docs.
>>>> Could you explain about that?
>>>> Thanks,
>>>> Shlomi
>>>>
>>>> On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
>>>>
>>>> What's your setting of log.flush.interval.messages on the broker?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
>>>>>
>>>> wrote:
>>>
>>>> Hi,
>>>>>> I am using 0.8.1.1, and I have hundreds of msec latency at best and
>>>>>>
>>>>> even
>>>>
>>>>> seconds at worst.
>>>>>> I have this latency both on production, (with peak load of 30K
>>>>>>
>>>>> msg/sec,
>>>
>>>> replication = 2 across 5 brokers, acks = 1),
>>>>>> and on the local windows machine using just one process for each of
>>>>>> producer, zookeeper, kafka, consumer.
>>>>>> Also tried batch.num.messages=1 and producer.type=sync on the local
>>>>>>
>>>>> machine
>>>>>
>>>>>> but saw no improvement.
>>>>>> How can I push latency down to several millis, at least when running
>>>>>>
>>>>> local?
>>>>>
>>>>>> Thanks,
>>>>>> Shlomi
>>>>>>
>>>>>>
>
>
Re: latency - how to reduce?
Posted by Andrew Ehrlich <an...@aehrlich.com>.
What instance types are you using in EC2? Are the drives EBS?
On 1/5/15 11:06 PM, Shlomi Hazan wrote:
> Will do. What did you have in mind? just write a big file to disk and
> measure the time it took to write? maybe also read back? using specific
> API's?
> Apart from the local Win machine case, are you aware of any issues with
> Amazon EC2 instances that may be causing that same latency in production?
> Thanks,
> Shlomi
>
> On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
>
>> Not setting "log.flush.interval.messages" is good since the default gives
>> the best latency. Could you do some basic I/O testing on the local FS in
>> your windows machine to make sure the I/O latency is ok?
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
>>
>>> Happy new year!
>>> I did not set "log.flush.interval.messages".
>>> I also could not find a default value in the docs.
>>> Could you explain about that?
>>> Thanks,
>>> Shlomi
>>>
>>> On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
>>>
>>>> What's your setting of log.flush.interval.messages on the broker?
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
>> wrote:
>>>>> Hi,
>>>>> I am using 0.8.1.1, and I have hundreds of msec latency at best and
>>> even
>>>>> seconds at worst.
>>>>> I have this latency both on production, (with peak load of 30K
>> msg/sec,
>>>>> replication = 2 across 5 brokers, acks = 1),
>>>>> and on the local windows machine using just one process for each of
>>>>> producer, zookeeper, kafka, consumer.
>>>>> Also tried batch.num.messages=1 and producer.type=sync on the local
>>>> machine
>>>>> but saw no improvement.
>>>>> How can I push latency down to several millis, at least when running
>>>> local?
>>>>> Thanks,
>>>>> Shlomi
>>>>>
Re: latency - how to reduce?
Posted by Shlomi Hazan <sh...@viber.com>.
Will do. What did you have in mind? just write a big file to disk and
measure the time it took to write? maybe also read back? using specific
API's?
Apart from the local Win machine case, are you aware of any issues with
Amazon EC2 instances that may be causing that same latency in production?
Thanks,
Shlomi
On Tue, Jan 6, 2015 at 4:04 AM, Jun Rao <ju...@confluent.io> wrote:
> Not setting "log.flush.interval.messages" is good since the default gives
> the best latency. Could you do some basic I/O testing on the local FS in
> your windows machine to make sure the I/O latency is ok?
>
> Thanks,
>
> Jun
>
> On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
>
> > Happy new year!
> > I did not set "log.flush.interval.messages".
> > I also could not find a default value in the docs.
> > Could you explain about that?
> > Thanks,
> > Shlomi
> >
> > On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > What's your setting of log.flush.interval.messages on the broker?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com>
> wrote:
> > >
> > > > Hi,
> > > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> > even
> > > > seconds at worst.
> > > > I have this latency both on production, (with peak load of 30K
> msg/sec,
> > > > replication = 2 across 5 brokers, acks = 1),
> > > > and on the local windows machine using just one process for each of
> > > > producer, zookeeper, kafka, consumer.
> > > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > > machine
> > > > but saw no improvement.
> > > > How can I push latency down to several millis, at least when running
> > > local?
> > > > Thanks,
> > > > Shlomi
> > > >
> > >
> >
>
Re: latency - how to reduce?
Posted by Jun Rao <ju...@confluent.io>.
Not setting "log.flush.interval.messages" is good since the default gives
the best latency. Could you do some basic I/O testing on the local FS in
your windows machine to make sure the I/O latency is ok?
Thanks,
Jun
On Thu, Jan 1, 2015 at 1:40 AM, Shlomi Hazan <sh...@viber.com> wrote:
> Happy new year!
> I did not set "log.flush.interval.messages".
> I also could not find a default value in the docs.
> Could you explain about that?
> Thanks,
> Shlomi
>
> On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > What's your setting of log.flush.interval.messages on the broker?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com> wrote:
> >
> > > Hi,
> > > I am using 0.8.1.1, and I have hundreds of msec latency at best and
> even
> > > seconds at worst.
> > > I have this latency both on production, (with peak load of 30K msg/sec,
> > > replication = 2 across 5 brokers, acks = 1),
> > > and on the local windows machine using just one process for each of
> > > producer, zookeeper, kafka, consumer.
> > > Also tried batch.num.messages=1 and producer.type=sync on the local
> > machine
> > > but saw no improvement.
> > > How can I push latency down to several millis, at least when running
> > local?
> > > Thanks,
> > > Shlomi
> > >
> >
>
Re: latency - how to reduce?
Posted by Shlomi Hazan <sh...@viber.com>.
Happy new year!
I did not set "log.flush.interval.messages".
I also could not find a default value in the docs.
Could you explain about that?
Thanks,
Shlomi
On Thu, Jan 1, 2015 at 2:20 AM, Jun Rao <ju...@confluent.io> wrote:
> What's your setting of log.flush.interval.messages on the broker?
>
> Thanks,
>
> Jun
>
> On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com> wrote:
>
> > Hi,
> > I am using 0.8.1.1, and I have hundreds of msec latency at best and even
> > seconds at worst.
> > I have this latency both on production, (with peak load of 30K msg/sec,
> > replication = 2 across 5 brokers, acks = 1),
> > and on the local windows machine using just one process for each of
> > producer, zookeeper, kafka, consumer.
> > Also tried batch.num.messages=1 and producer.type=sync on the local
> machine
> > but saw no improvement.
> > How can I push latency down to several millis, at least when running
> local?
> > Thanks,
> > Shlomi
> >
>
Re: latency - how to reduce?
Posted by Jun Rao <ju...@confluent.io>.
What's your setting of log.flush.interval.messages on the broker?
Thanks,
Jun
On Mon, Dec 29, 2014 at 3:26 AM, Shlomi Hazan <sh...@viber.com> wrote:
> Hi,
> I am using 0.8.1.1, and I have hundreds of msec latency at best and even
> seconds at worst.
> I have this latency both on production, (with peak load of 30K msg/sec,
> replication = 2 across 5 brokers, acks = 1),
> and on the local windows machine using just one process for each of
> producer, zookeeper, kafka, consumer.
> Also tried batch.num.messages=1 and producer.type=sync on the local machine
> but saw no improvement.
> How can I push latency down to several millis, at least when running local?
> Thanks,
> Shlomi
>