You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Maung Than <ma...@apple.com> on 2014/05/27 23:41:38 UTC

Question on output of kafka-producer-perf-test.sh

Hi All, 

We are seeing total data sent below is the same with or without compression. 
Is it always raw data? 
If so is there a way we can get the compressed volume? 

start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
[2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72, 52.4285, 5000000, 130893.4789


-bash-4.1$ /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic imessage --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
[2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
[2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid (kafka.utils.VerifiableProperties)
2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72, 49.6102, 5000000, 123857.4153

Thanks,
Maung

Re: Question on output of kafka-producer-perf-test.sh

Posted by Maung Than <ma...@apple.com>.
As you advised, I wrote a script to calculate the file sizes on each broker and run the tests again. 

We are finding that if we are run this tool ( kafka-producer-perf-test.sh ) while firing from the producers the data on the broker actually bloomed to almost double.

It only happens with no compression mode. We tested multiple times and it would be nice if some one can validate this as we can’t think of anything else. 

To make sure each test is equal — we wipe out all the data / metadata directories for all brokers and zookeeper and start form scratch each time. 

( One topic, five brokers, five partitions, no replication factor, two producer machine with 10 threads each) 


--The performance test command used is : /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh --broker-list vp21q12ic-hpaj020921:9092 --messages 200000000 --topic imessage --threads 20 --message-size 405 --batch-size 1000 --compression-codec 2

I now suspect the TimeOutException with Gzip could be caused by this tool running while we are sending the data. 

Thanks,
Maung

On May 28, 2014, at 9:37 PM, Jun Rao <ju...@gmail.com> wrote:

> If you start with an empty topic, the log size on the broker gives you the
> total bytes sent.
> 
> Thanks,
> 
> Jun
> 
> 
> On Wed, May 28, 2014 at 10:38 AM, Maung Than <ma...@apple.com> wrote:
> 
>> 
>>>>>> The bytes-in-rate reported on the broker is the post-compression rate.
>> 
>> Based on that fact, I am trying to calculate the volume by multiplying
>> duration and rate, i.e,  (end.time - start.time) X MB.sec ,
>> but it does not give me the total.data.sent.in.MB.I put the duration in
>> the brackets below:
>> 
>> Any thoughts?
>> 
>> 
>> 2014-05-27 14:10:37:347,    2014-05-27 14:11:15:546, 0,  420,  1000,
>> 2002.72,   52.4285,  5000000,  130893.4789  ( 34 sec )
>> 
>> 
>> 2014-05-27 13:33:31:493,    2014-05-27 13:34:11:862, 2,  420,  1000,
>> 2002.72,   49.6102,  5000000,  123857.4153  ( 40 sec )
>> 
>> Thanks,
>> Maung
>> 
>> 
>> On May 28, 2014, at 7:37 AM, Jun Rao <ju...@gmail.com> wrote:
>> 
>>> The bytes-in-rate reported on the broker is the post-compression rate.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> 
>>> On Tue, May 27, 2014 at 9:16 PM, Maung Than <ma...@apple.com>
>> wrote:
>>> 
>>>> Any idea when and which release that could be included in?
>>>> 
>>>> We would like to have it sooner and can we do something about it?
>>>> 
>>>> Thanks,
>>>> Maung
>>>> 
>>>> On May 27, 2014, at 5:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>>>> 
>>>>> Maung,
>>>>> 
>>>>> Yes, the throughput computed here is based on pre-compression bytes. In
>>>> the
>>>>> old producer we do not have a metric exposing the compression ratio, in
>>>> the
>>>>> new producer available in current trunk we can easily add that metric.
>> I
>>>>> have just created a ticket for this a moment ago.
>>>>> 
>>>>> https://issues.apache.org/jira/browse/KAFKA-1472
>>>>> 
>>>>> Guozhang
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com>
>>>> wrote:
>>>>> 
>>>>>> 
>>>>>> Hi All,
>>>>>> 
>>>>>> We are seeing total data sent below is the same with or without
>>>>>> compression.
>>>>>> Is it always raw data?
>>>>>> If so is there a way we can get the compressed volume?
>>>>>> 
>>>>>> start.time, end.time, compression, message.size, batch.size,
>>>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>> further
>>>>>> details.
>>>>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000,
>> 2002.72,
>>>>>> 52.4285, 5000000, 130893.4789
>>>>>> 
>>>>>> 
>>>>>> -bash-4.1$
>>>>>> 
>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
>>>>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
>>>> imessage
>>>>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
>>>>>> start.time, end.time, compression, message.size, batch.size,
>>>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>>>> further
>>>>>> details.
>>>>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not
>> valid
>>>>>> (kafka.utils.VerifiableProperties)
>>>>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000,
>> 2002.72,
>>>>>> 49.6102, 5000000, 123857.4153
>>>>>> 
>>>>>> Thanks,
>>>>>> Maung
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> -- Guozhang
>>>> 
>>>> 
>> 
>>

Re: Question on output of kafka-producer-perf-test.sh

Posted by Jun Rao <ju...@gmail.com>.
If you start with an empty topic, the log size on the broker gives you the
total bytes sent.

Thanks,

Jun


On Wed, May 28, 2014 at 10:38 AM, Maung Than <ma...@apple.com> wrote:

>
> >>>> The bytes-in-rate reported on the broker is the post-compression rate.
>
> Based on that fact, I am trying to calculate the volume by multiplying
> duration and rate, i.e,  (end.time - start.time) X MB.sec ,
> but it does not give me the total.data.sent.in.MB.I put the duration in
> the brackets below:
>
> Any thoughts?
>
>
> 2014-05-27 14:10:37:347,    2014-05-27 14:11:15:546, 0,  420,  1000,
>  2002.72,   52.4285,  5000000,  130893.4789  ( 34 sec )
>
>
> 2014-05-27 13:33:31:493,    2014-05-27 13:34:11:862, 2,  420,  1000,
>  2002.72,   49.6102,  5000000,  123857.4153  ( 40 sec )
>
> Thanks,
> Maung
>
>
> On May 28, 2014, at 7:37 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > The bytes-in-rate reported on the broker is the post-compression rate.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, May 27, 2014 at 9:16 PM, Maung Than <ma...@apple.com>
> wrote:
> >
> >> Any idea when and which release that could be included in?
> >>
> >> We would like to have it sooner and can we do something about it?
> >>
> >> Thanks,
> >> Maung
> >>
> >> On May 27, 2014, at 5:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
> >>
> >>> Maung,
> >>>
> >>> Yes, the throughput computed here is based on pre-compression bytes. In
> >> the
> >>> old producer we do not have a metric exposing the compression ratio, in
> >> the
> >>> new producer available in current trunk we can easily add that metric.
> I
> >>> have just created a ticket for this a moment ago.
> >>>
> >>> https://issues.apache.org/jira/browse/KAFKA-1472
> >>>
> >>> Guozhang
> >>>
> >>>
> >>>
> >>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com>
> >> wrote:
> >>>
> >>>>
> >>>> Hi All,
> >>>>
> >>>> We are seeing total data sent below is the same with or without
> >>>> compression.
> >>>> Is it always raw data?
> >>>> If so is there a way we can get the compressed volume?
> >>>>
> >>>> start.time, end.time, compression, message.size, batch.size,
> >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> >>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> >> further
> >>>> details.
> >>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000,
> 2002.72,
> >>>> 52.4285, 5000000, 130893.4789
> >>>>
> >>>>
> >>>> -bash-4.1$
> >>>>
> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
> >>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
> >> imessage
> >>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
> >>>> start.time, end.time, compression, message.size, batch.size,
> >>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> >>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> >> further
> >>>> details.
> >>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not
> valid
> >>>> (kafka.utils.VerifiableProperties)
> >>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000,
> 2002.72,
> >>>> 49.6102, 5000000, 123857.4153
> >>>>
> >>>> Thanks,
> >>>> Maung
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>
> >>
>
>

Re: Question on output of kafka-producer-perf-test.sh

Posted by Maung Than <ma...@apple.com>.
>>>> The bytes-in-rate reported on the broker is the post-compression rate.

Based on that fact, I am trying to calculate the volume by multiplying duration and rate, i.e,  (end.time - start.time) X MB.sec , 
but it does not give me the total.data.sent.in.MB.I put the duration in the brackets below: 

Any thoughts? 


2014-05-27 14:10:37:347,    2014-05-27 14:11:15:546, 0,  420,  1000,  2002.72,   52.4285,  5000000,  130893.4789  ( 34 sec )


2014-05-27 13:33:31:493,    2014-05-27 13:34:11:862, 2,  420,  1000,  2002.72,   49.6102,  5000000,  123857.4153  ( 40 sec ) 

Thanks,
Maung


On May 28, 2014, at 7:37 AM, Jun Rao <ju...@gmail.com> wrote:

> The bytes-in-rate reported on the broker is the post-compression rate.
> 
> Thanks,
> 
> Jun
> 
> 
> On Tue, May 27, 2014 at 9:16 PM, Maung Than <ma...@apple.com> wrote:
> 
>> Any idea when and which release that could be included in?
>> 
>> We would like to have it sooner and can we do something about it?
>> 
>> Thanks,
>> Maung
>> 
>> On May 27, 2014, at 5:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>> 
>>> Maung,
>>> 
>>> Yes, the throughput computed here is based on pre-compression bytes. In
>> the
>>> old producer we do not have a metric exposing the compression ratio, in
>> the
>>> new producer available in current trunk we can easily add that metric. I
>>> have just created a ticket for this a moment ago.
>>> 
>>> https://issues.apache.org/jira/browse/KAFKA-1472
>>> 
>>> Guozhang
>>> 
>>> 
>>> 
>>> On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com>
>> wrote:
>>> 
>>>> 
>>>> Hi All,
>>>> 
>>>> We are seeing total data sent below is the same with or without
>>>> compression.
>>>> Is it always raw data?
>>>> If so is there a way we can get the compressed volume?
>>>> 
>>>> start.time, end.time, compression, message.size, batch.size,
>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further
>>>> details.
>>>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
>>>> 52.4285, 5000000, 130893.4789
>>>> 
>>>> 
>>>> -bash-4.1$
>>>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
>>>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
>> imessage
>>>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
>>>> start.time, end.time, compression, message.size, batch.size,
>>>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>>>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further
>>>> details.
>>>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid
>>>> (kafka.utils.VerifiableProperties)
>>>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72,
>>>> 49.6102, 5000000, 123857.4153
>>>> 
>>>> Thanks,
>>>> Maung
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> -- Guozhang
>> 
>>

Re: Question on output of kafka-producer-perf-test.sh

Posted by Jun Rao <ju...@gmail.com>.
The bytes-in-rate reported on the broker is the post-compression rate.

Thanks,

Jun


On Tue, May 27, 2014 at 9:16 PM, Maung Than <ma...@apple.com> wrote:

> Any idea when and which release that could be included in?
>
> We would like to have it sooner and can we do something about it?
>
> Thanks,
> Maung
>
> On May 27, 2014, at 5:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Maung,
> >
> > Yes, the throughput computed here is based on pre-compression bytes. In
> the
> > old producer we do not have a metric exposing the compression ratio, in
> the
> > new producer available in current trunk we can easily add that metric. I
> > have just created a ticket for this a moment ago.
> >
> > https://issues.apache.org/jira/browse/KAFKA-1472
> >
> > Guozhang
> >
> >
> >
> > On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com>
> wrote:
> >
> >>
> >> Hi All,
> >>
> >> We are seeing total data sent below is the same with or without
> >> compression.
> >> Is it always raw data?
> >> If so is there a way we can get the compressed volume?
> >>
> >> start.time, end.time, compression, message.size, batch.size,
> >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> >> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> >> details.
> >> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
> >> 52.4285, 5000000, 130893.4789
> >>
> >>
> >> -bash-4.1$
> >> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
> >> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
> imessage
> >> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
> >> start.time, end.time, compression, message.size, batch.size,
> >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> >> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> >> details.
> >> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72,
> >> 49.6102, 5000000, 123857.4153
> >>
> >> Thanks,
> >> Maung
> >>
> >
> >
> >
> > --
> > -- Guozhang
>
>

Re: Question on output of kafka-producer-perf-test.sh

Posted by Guozhang Wang <wa...@gmail.com>.
It would be probably included in the 0.8.2 release. I have looked into its
implementation and it should be in fact a quite simple jira, so we would
label it as "newbie" and you are more welcomed to try it on.

Guozhang


On Tue, May 27, 2014 at 9:16 PM, Maung Than <ma...@apple.com> wrote:

> Any idea when and which release that could be included in?
>
> We would like to have it sooner and can we do something about it?
>
> Thanks,
> Maung
>
> On May 27, 2014, at 5:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Maung,
> >
> > Yes, the throughput computed here is based on pre-compression bytes. In
> the
> > old producer we do not have a metric exposing the compression ratio, in
> the
> > new producer available in current trunk we can easily add that metric. I
> > have just created a ticket for this a moment ago.
> >
> > https://issues.apache.org/jira/browse/KAFKA-1472
> >
> > Guozhang
> >
> >
> >
> > On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com>
> wrote:
> >
> >>
> >> Hi All,
> >>
> >> We are seeing total data sent below is the same with or without
> >> compression.
> >> Is it always raw data?
> >> If so is there a way we can get the compressed volume?
> >>
> >> start.time, end.time, compression, message.size, batch.size,
> >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> >> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> >> details.
> >> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
> >> 52.4285, 5000000, 130893.4789
> >>
> >>
> >> -bash-4.1$
> >> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
> >> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic
> imessage
> >> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
> >> start.time, end.time, compression, message.size, batch.size,
> >> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> >> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> >> SLF4J: Defaulting to no-operation (NOP) logger implementation
> >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> >> details.
> >> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid
> >> (kafka.utils.VerifiableProperties)
> >> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72,
> >> 49.6102, 5000000, 123857.4153
> >>
> >> Thanks,
> >> Maung
> >>
> >
> >
> >
> > --
> > -- Guozhang
>
>


-- 
-- Guozhang

Re: Question on output of kafka-producer-perf-test.sh

Posted by Maung Than <ma...@apple.com>.
Any idea when and which release that could be included in?

We would like to have it sooner and can we do something about it?

Thanks,
Maung

On May 27, 2014, at 5:10 PM, Guozhang Wang <wa...@gmail.com> wrote:

> Maung,
> 
> Yes, the throughput computed here is based on pre-compression bytes. In the
> old producer we do not have a metric exposing the compression ratio, in the
> new producer available in current trunk we can easily add that metric. I
> have just created a ticket for this a moment ago.
> 
> https://issues.apache.org/jira/browse/KAFKA-1472
> 
> Guozhang
> 
> 
> 
> On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com> wrote:
> 
>> 
>> Hi All,
>> 
>> We are seeing total data sent below is the same with or without
>> compression.
>> Is it always raw data?
>> If so is there a way we can get the compressed volume?
>> 
>> start.time, end.time, compression, message.size, batch.size,
>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
>> details.
>> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
>> 52.4285, 5000000, 130893.4789
>> 
>> 
>> -bash-4.1$
>> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
>> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic imessage
>> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
>> start.time, end.time, compression, message.size, batch.size,
>> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
>> details.
>> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid
>> (kafka.utils.VerifiableProperties)
>> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72,
>> 49.6102, 5000000, 123857.4153
>> 
>> Thanks,
>> Maung
>> 
> 
> 
> 
> -- 
> -- Guozhang

Re: Question on output of kafka-producer-perf-test.sh

Posted by Guozhang Wang <wa...@gmail.com>.
Maung,

Yes, the throughput computed here is based on pre-compression bytes. In the
old producer we do not have a metric exposing the compression ratio, in the
new producer available in current trunk we can easily add that metric. I
have just created a ticket for this a moment ago.

https://issues.apache.org/jira/browse/KAFKA-1472

Guozhang



On Tue, May 27, 2014 at 2:41 PM, Maung Than <ma...@apple.com> wrote:

>
> Hi All,
>
> We are seeing total data sent below is the same with or without
> compression.
> Is it always raw data?
> If so is there a way we can get the compressed volume?
>
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> [2014-05-27 14:10:37,471] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> [2014-05-27 14:10:37,547] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> [2014-05-27 14:10:37,549] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> [2014-05-27 14:10:37,552] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> [2014-05-27 14:10:37,560] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> 2014-05-27 14:10:37:347, 2014-05-27 14:11:15:546, 0, 420, 1000, 2002.72,
> 52.4285, 5000000, 130893.4789
>
>
> -bash-4.1$
> /Users/worun/kafkabuild/kafka-0.8.1-src/bin/kafka-producer-perf-test.sh
> --broker-list vp21q12ic-hpaj020921:9092 --messages 5000000 --topic imessage
> --threads 5 --message-size 420 --batch-size 1000 --compression-codec 2
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> [2014-05-27 13:33:31,616] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> [2014-05-27 13:33:31,692] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> [2014-05-27 13:33:31,694] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> [2014-05-27 13:33:31,697] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> [2014-05-27 13:33:31,705] WARN Property reconnect.interval is not valid
> (kafka.utils.VerifiableProperties)
> 2014-05-27 13:33:31:493, 2014-05-27 13:34:11:862, 2, 420, 1000, 2002.72,
> 49.6102, 5000000, 123857.4153
>
> Thanks,
> Maung
>



-- 
-- Guozhang