You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jon Yeargers <jo...@cedexis.com> on 2016/12/16 20:00:31 UTC
effect of high IOWait on KStream app?
Looking for reasons why my installations seem to be generating so many
issues:
Starting an app which is
stream->aggregate->filter->foreach
While it's running the system in question (AWS) averages >10% IOWait with
spikes to 60-70%. The CPU load is in the range of 3/2/1 (8 core machine w/
16G RAM)
Could this IO delay be causing problems? The 'kafka-streams' folder is on a
separate EBS optimized volume with 2500 dedicated IOPs.
Re: effect of high IOWait on KStream app?
Posted by Jon Yeargers <jo...@cedexis.com>.
Could this delay be contributing to the various issues Im seeing?
- extended / repeating rebalances
- broker connection drops
On Sat, Dec 17, 2016 at 10:27 AM, Eno Thereska <en...@gmail.com>
wrote:
> Yeah, the numbers for the streams tests seem to be low. For reference,
> here is what I get when I run it on my laptop, with Kafka co-located
> (Macbook pro, 16GB, SSD). These are rounded up with no decimal places:
>
> > Producer Performance [MB/sec write]: 40
> > Consumer Performance [MB/sec read]: 126
>
> > Streams Performance [MB/sec read]: 81
> > Streams Performance [MB/sec read+write]: 45
> > Streams Performance [MB/sec read+store]: 22
> > Streams KStreamKTable LeftJoin Performance [MB/s joined]: 51
> > Streams KStreamKStream LeftJoin Performance [MB/s joined]: 11
> > Streams KTableKTable LeftJoin Performance [MB/s joined]: 12
>
> I haven't tried this on AWS unfortunately so I don't know what to expect
> there.
>
> Eno
>
>
> > On 17 Dec 2016, at 15:39, Jon Yeargers <jo...@cedexis.com> wrote:
> >
> > stateDir=/tmp/kafka-streams-simple-benchmark
> >
> > numRecords=10000000
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> >
> > SLF4J: Found binding in
> > [jar:file:/home/ec2-user/kafka/streams/build/dependant-
> libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> >
> > SLF4J: Found binding in
> > [jar:file:/home/ec2-user/kafka/tools/build/dependant-
> libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/
> StaticLoggerBinder.class]
> >
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> >
> > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> >
> > [2016-12-17 15:26:15,011] WARN Error while fetching metadata with
> > correlation id 1 : {simpleBenchmarkSourceTopic=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Producer Performance [MB/sec write]: 87.52977203130317
> >
> > Consumer Performance [MB/sec read]: 88.05408180729077
> >
> > Streams Performance [MB/sec read]: 23.306380376435413
> >
> > [2016-12-17 15:27:22,722] WARN Error while fetching metadata with
> > correlation id 1 : {simpleBenchmarkSinkTopic=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Streams Performance [MB/sec read+write]: 20.37099360560648
> >
> > Streams Performance [MB/sec read+store]: 11.918550778354337
> >
> > Initializing kStreamTopic joinSourceTopic1kStreamKTable
> >
> > [2016-12-17 15:29:39,597] WARN Error while fetching metadata with
> > correlation id 1 : {joinSourceTopic1kStreamKTable=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Initializing kTableTopic joinSourceTopic2kStreamKTable
> >
> > [2016-12-17 15:29:50,589] WARN Error while fetching metadata with
> > correlation id 1 : {joinSourceTopic2kStreamKTable=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Streams KStreamKTable LeftJoin Performance [MB/s joined]:
> 14.690136622553428
> >
> > Initializing kStreamTopic joinSourceTopic1kStreamKStream
> >
> > [2016-12-17 15:31:12,583] WARN Error while fetching metadata with
> > correlation id 1 : {joinSourceTopic1kStreamKStream=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Initializing kStreamTopic joinSourceTopic2kStreamKStream
> >
> > [2016-12-17 15:31:23,534] WARN Error while fetching metadata with
> > correlation id 1 : {joinSourceTopic2kStreamKStream=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Streams KStreamKStream LeftJoin Performance [MB/s joined]:
> 8.647640177490924
> >
> > Initializing kTableTopic joinSourceTopic1kTableKTable
> >
> > [2016-12-17 15:33:34,586] WARN Error while fetching metadata with
> > correlation id 1 : {joinSourceTopic1kTableKTable=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Initializing kTableTopic joinSourceTopic2kTableKTable
> >
> > [2016-12-17 15:33:45,520] WARN Error while fetching metadata with
> > correlation id 1 : {joinSourceTopic2kTableKTable=LEADER_NOT_AVAILABLE}
> > (org.apache.kafka.clients.NetworkClient:709)
> >
> > Streams KTableKTable LeftJoin Performance [MB/s joined]:
> 6.530348031376133
> >
> > On Sat, Dec 17, 2016 at 6:39 AM, Jon Yeargers <jo...@cedexis.com>
> > wrote:
> >
> >> I'd be happy to but the AWS AMI (default) i'm using is fighting this at
> >> every turn. Will keep trying.
> >>
> >> On Sat, Dec 17, 2016 at 2:46 AM, Eno Thereska <en...@gmail.com>
> >> wrote:
> >>
> >>> Jon,
> >>>
> >>> It's hard to tell. Would you be willing to run a simple benchmark and
> >>> report back the numbers? The benchmark is called SimpleBenchmark.java,
> it's
> >>> included with the source, and it will start a couple of streams apps.
> It
> >>> requires a ZK and a broker to be up. Then you run it:
> >>> org.apache.kafka.streams.perf.SimpleBenchmark
> <zookeeperHostPortString>
> >>> <brokerHostPortString>.
> >>>
> >>> Thanks
> >>> Eno
> >>>> On 16 Dec 2016, at 20:00, Jon Yeargers <jo...@cedexis.com>
> >>> wrote:
> >>>>
> >>>> Looking for reasons why my installations seem to be generating so many
> >>>> issues:
> >>>>
> >>>> Starting an app which is
> >>>>
> >>>> stream->aggregate->filter->foreach
> >>>>
> >>>> While it's running the system in question (AWS) averages >10% IOWait
> >>> with
> >>>> spikes to 60-70%. The CPU load is in the range of 3/2/1 (8 core
> >>> machine w/
> >>>> 16G RAM)
> >>>>
> >>>> Could this IO delay be causing problems? The 'kafka-streams' folder is
> >>> on a
> >>>> separate EBS optimized volume with 2500 dedicated IOPs.
> >>>
> >>>
> >>
>
>
Re: effect of high IOWait on KStream app?
Posted by Eno Thereska <en...@gmail.com>.
Yeah, the numbers for the streams tests seem to be low. For reference, here is what I get when I run it on my laptop, with Kafka co-located (Macbook pro, 16GB, SSD). These are rounded up with no decimal places:
> Producer Performance [MB/sec write]: 40
> Consumer Performance [MB/sec read]: 126
> Streams Performance [MB/sec read]: 81
> Streams Performance [MB/sec read+write]: 45
> Streams Performance [MB/sec read+store]: 22
> Streams KStreamKTable LeftJoin Performance [MB/s joined]: 51
> Streams KStreamKStream LeftJoin Performance [MB/s joined]: 11
> Streams KTableKTable LeftJoin Performance [MB/s joined]: 12
I haven't tried this on AWS unfortunately so I don't know what to expect there.
Eno
> On 17 Dec 2016, at 15:39, Jon Yeargers <jo...@cedexis.com> wrote:
>
> stateDir=/tmp/kafka-streams-simple-benchmark
>
> numRecords=10000000
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in
> [jar:file:/home/ec2-user/kafka/streams/build/dependant-libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:file:/home/ec2-user/kafka/tools/build/dependant-libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> [2016-12-17 15:26:15,011] WARN Error while fetching metadata with
> correlation id 1 : {simpleBenchmarkSourceTopic=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Producer Performance [MB/sec write]: 87.52977203130317
>
> Consumer Performance [MB/sec read]: 88.05408180729077
>
> Streams Performance [MB/sec read]: 23.306380376435413
>
> [2016-12-17 15:27:22,722] WARN Error while fetching metadata with
> correlation id 1 : {simpleBenchmarkSinkTopic=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Streams Performance [MB/sec read+write]: 20.37099360560648
>
> Streams Performance [MB/sec read+store]: 11.918550778354337
>
> Initializing kStreamTopic joinSourceTopic1kStreamKTable
>
> [2016-12-17 15:29:39,597] WARN Error while fetching metadata with
> correlation id 1 : {joinSourceTopic1kStreamKTable=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Initializing kTableTopic joinSourceTopic2kStreamKTable
>
> [2016-12-17 15:29:50,589] WARN Error while fetching metadata with
> correlation id 1 : {joinSourceTopic2kStreamKTable=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Streams KStreamKTable LeftJoin Performance [MB/s joined]: 14.690136622553428
>
> Initializing kStreamTopic joinSourceTopic1kStreamKStream
>
> [2016-12-17 15:31:12,583] WARN Error while fetching metadata with
> correlation id 1 : {joinSourceTopic1kStreamKStream=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Initializing kStreamTopic joinSourceTopic2kStreamKStream
>
> [2016-12-17 15:31:23,534] WARN Error while fetching metadata with
> correlation id 1 : {joinSourceTopic2kStreamKStream=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Streams KStreamKStream LeftJoin Performance [MB/s joined]: 8.647640177490924
>
> Initializing kTableTopic joinSourceTopic1kTableKTable
>
> [2016-12-17 15:33:34,586] WARN Error while fetching metadata with
> correlation id 1 : {joinSourceTopic1kTableKTable=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Initializing kTableTopic joinSourceTopic2kTableKTable
>
> [2016-12-17 15:33:45,520] WARN Error while fetching metadata with
> correlation id 1 : {joinSourceTopic2kTableKTable=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient:709)
>
> Streams KTableKTable LeftJoin Performance [MB/s joined]: 6.530348031376133
>
> On Sat, Dec 17, 2016 at 6:39 AM, Jon Yeargers <jo...@cedexis.com>
> wrote:
>
>> I'd be happy to but the AWS AMI (default) i'm using is fighting this at
>> every turn. Will keep trying.
>>
>> On Sat, Dec 17, 2016 at 2:46 AM, Eno Thereska <en...@gmail.com>
>> wrote:
>>
>>> Jon,
>>>
>>> It's hard to tell. Would you be willing to run a simple benchmark and
>>> report back the numbers? The benchmark is called SimpleBenchmark.java, it's
>>> included with the source, and it will start a couple of streams apps. It
>>> requires a ZK and a broker to be up. Then you run it:
>>> org.apache.kafka.streams.perf.SimpleBenchmark <zookeeperHostPortString>
>>> <brokerHostPortString>.
>>>
>>> Thanks
>>> Eno
>>>> On 16 Dec 2016, at 20:00, Jon Yeargers <jo...@cedexis.com>
>>> wrote:
>>>>
>>>> Looking for reasons why my installations seem to be generating so many
>>>> issues:
>>>>
>>>> Starting an app which is
>>>>
>>>> stream->aggregate->filter->foreach
>>>>
>>>> While it's running the system in question (AWS) averages >10% IOWait
>>> with
>>>> spikes to 60-70%. The CPU load is in the range of 3/2/1 (8 core
>>> machine w/
>>>> 16G RAM)
>>>>
>>>> Could this IO delay be causing problems? The 'kafka-streams' folder is
>>> on a
>>>> separate EBS optimized volume with 2500 dedicated IOPs.
>>>
>>>
>>
Re: effect of high IOWait on KStream app?
Posted by Jon Yeargers <jo...@cedexis.com>.
stateDir=/tmp/kafka-streams-simple-benchmark
numRecords=10000000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/ec2-user/kafka/streams/build/dependant-libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/ec2-user/kafka/tools/build/dependant-libs-2.10.6/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2016-12-17 15:26:15,011] WARN Error while fetching metadata with
correlation id 1 : {simpleBenchmarkSourceTopic=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Producer Performance [MB/sec write]: 87.52977203130317
Consumer Performance [MB/sec read]: 88.05408180729077
Streams Performance [MB/sec read]: 23.306380376435413
[2016-12-17 15:27:22,722] WARN Error while fetching metadata with
correlation id 1 : {simpleBenchmarkSinkTopic=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Streams Performance [MB/sec read+write]: 20.37099360560648
Streams Performance [MB/sec read+store]: 11.918550778354337
Initializing kStreamTopic joinSourceTopic1kStreamKTable
[2016-12-17 15:29:39,597] WARN Error while fetching metadata with
correlation id 1 : {joinSourceTopic1kStreamKTable=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Initializing kTableTopic joinSourceTopic2kStreamKTable
[2016-12-17 15:29:50,589] WARN Error while fetching metadata with
correlation id 1 : {joinSourceTopic2kStreamKTable=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Streams KStreamKTable LeftJoin Performance [MB/s joined]: 14.690136622553428
Initializing kStreamTopic joinSourceTopic1kStreamKStream
[2016-12-17 15:31:12,583] WARN Error while fetching metadata with
correlation id 1 : {joinSourceTopic1kStreamKStream=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Initializing kStreamTopic joinSourceTopic2kStreamKStream
[2016-12-17 15:31:23,534] WARN Error while fetching metadata with
correlation id 1 : {joinSourceTopic2kStreamKStream=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Streams KStreamKStream LeftJoin Performance [MB/s joined]: 8.647640177490924
Initializing kTableTopic joinSourceTopic1kTableKTable
[2016-12-17 15:33:34,586] WARN Error while fetching metadata with
correlation id 1 : {joinSourceTopic1kTableKTable=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Initializing kTableTopic joinSourceTopic2kTableKTable
[2016-12-17 15:33:45,520] WARN Error while fetching metadata with
correlation id 1 : {joinSourceTopic2kTableKTable=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient:709)
Streams KTableKTable LeftJoin Performance [MB/s joined]: 6.530348031376133
On Sat, Dec 17, 2016 at 6:39 AM, Jon Yeargers <jo...@cedexis.com>
wrote:
> I'd be happy to but the AWS AMI (default) i'm using is fighting this at
> every turn. Will keep trying.
>
> On Sat, Dec 17, 2016 at 2:46 AM, Eno Thereska <en...@gmail.com>
> wrote:
>
>> Jon,
>>
>> It's hard to tell. Would you be willing to run a simple benchmark and
>> report back the numbers? The benchmark is called SimpleBenchmark.java, it's
>> included with the source, and it will start a couple of streams apps. It
>> requires a ZK and a broker to be up. Then you run it:
>> org.apache.kafka.streams.perf.SimpleBenchmark <zookeeperHostPortString>
>> <brokerHostPortString>.
>>
>> Thanks
>> Eno
>> > On 16 Dec 2016, at 20:00, Jon Yeargers <jo...@cedexis.com>
>> wrote:
>> >
>> > Looking for reasons why my installations seem to be generating so many
>> > issues:
>> >
>> > Starting an app which is
>> >
>> > stream->aggregate->filter->foreach
>> >
>> > While it's running the system in question (AWS) averages >10% IOWait
>> with
>> > spikes to 60-70%. The CPU load is in the range of 3/2/1 (8 core
>> machine w/
>> > 16G RAM)
>> >
>> > Could this IO delay be causing problems? The 'kafka-streams' folder is
>> on a
>> > separate EBS optimized volume with 2500 dedicated IOPs.
>>
>>
>
Re: effect of high IOWait on KStream app?
Posted by Jon Yeargers <jo...@cedexis.com>.
I'd be happy to but the AWS AMI (default) i'm using is fighting this at
every turn. Will keep trying.
On Sat, Dec 17, 2016 at 2:46 AM, Eno Thereska <en...@gmail.com>
wrote:
> Jon,
>
> It's hard to tell. Would you be willing to run a simple benchmark and
> report back the numbers? The benchmark is called SimpleBenchmark.java, it's
> included with the source, and it will start a couple of streams apps. It
> requires a ZK and a broker to be up. Then you run it:
> org.apache.kafka.streams.perf.SimpleBenchmark <zookeeperHostPortString>
> <brokerHostPortString>.
>
> Thanks
> Eno
> > On 16 Dec 2016, at 20:00, Jon Yeargers <jo...@cedexis.com> wrote:
> >
> > Looking for reasons why my installations seem to be generating so many
> > issues:
> >
> > Starting an app which is
> >
> > stream->aggregate->filter->foreach
> >
> > While it's running the system in question (AWS) averages >10% IOWait with
> > spikes to 60-70%. The CPU load is in the range of 3/2/1 (8 core machine
> w/
> > 16G RAM)
> >
> > Could this IO delay be causing problems? The 'kafka-streams' folder is
> on a
> > separate EBS optimized volume with 2500 dedicated IOPs.
>
>
Re: effect of high IOWait on KStream app?
Posted by Eno Thereska <en...@gmail.com>.
Jon,
It's hard to tell. Would you be willing to run a simple benchmark and report back the numbers? The benchmark is called SimpleBenchmark.java, it's included with the source, and it will start a couple of streams apps. It requires a ZK and a broker to be up. Then you run it:
org.apache.kafka.streams.perf.SimpleBenchmark <zookeeperHostPortString> <brokerHostPortString>.
Thanks
Eno
> On 16 Dec 2016, at 20:00, Jon Yeargers <jo...@cedexis.com> wrote:
>
> Looking for reasons why my installations seem to be generating so many
> issues:
>
> Starting an app which is
>
> stream->aggregate->filter->foreach
>
> While it's running the system in question (AWS) averages >10% IOWait with
> spikes to 60-70%. The CPU load is in the range of 3/2/1 (8 core machine w/
> 16G RAM)
>
> Could this IO delay be causing problems? The 'kafka-streams' folder is on a
> separate EBS optimized volume with 2500 dedicated IOPs.