You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Prabhjot Bharaj <pr...@gmail.com> on 2015/09/24 08:44:07 UTC
Re: Producer becomes slow over time
Hi,
I would like to dig deep into this issue. I've changed log4j.properties for
logging in ALL mode in all places. I am getting lost in the logs.
Any pointers would be welcome
Please let me know if you would need any information regarding this
Thanks,
Prabhjot
On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hello Folks,
>
> I've noticed that 2 producer machines, that I had configured, have become
> very slow over time
> They are giving 17-19 MB/s
>
> But, a producer that I setup today is giving 70MB/s as the write throughput
>
> If I see the contents of bin, libs, config directories, nothing is
> different in the files on any of the producer machines.
>
> Producer is running on the kafka machines itself
>
> Request your expertise
>
> Regards,
> Prabhjot
>
>
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Producer becomes slow over time
Posted by "Helleren, Erik" <Er...@cmegroup.com>.
Well, its possible that it is a networking issue which is why I asked for
results from a test from a machine that isn¹t on the kafka cluster. And
were all the producers run on the same member of the kafka cluster, or was
it spread out?
Also, for the topic you are testing with, how many partitions does it
have? And what is the replication factor? Also, how many kafka nodes do
you have?
On 9/29/15, 8:17 AM, "Prabhjot Bharaj" <pr...@gmail.com> wrote:
>Hi Erik,
>
>I've not tested it on a producer that is not a part of the kafka cluster.
>
>
>*Producer configuration on all machines:-*
>
>root@x.x.x.x:/a/kafka/config# cat producer.properties | egrep -v '^#|^$'
>
>metadata.broker.list=localhost:9092
>
>producer.type=sync
>
>compression.codec=none
>
>serializer.class=kafka.serializer.DefaultEncoder
>
>
>*Server configuration on all machines:-*
>
>root@y.y.y.y:/a/kafka/config# cat server.properties | egrep -v '^#|^$'
>
>broker.id=0
>
>port=9092
>
>num.network.threads=6
>
>num.io.threads=8
>
>socket.send.buffer.bytes=10485760
>
>socket.receive.buffer.bytes=10485760
>
>socket.request.max.bytes=104857600
>
>log.dirs=/tmp/kafka-logs
>
>num.partitions=1
>
>num.recovery.threads.per.data.dir=1
>
>log.retention.hours=168
>
>log.segment.bytes=1073741824
>
>log.retention.check.interval.ms=300000
>
>log.cleaner.enable=false
>
>zookeeper.connect=localhost:2181
>
>zookeeper.connection.timeout.ms=6000
>
>num.replica.fetchers=4
>
>*Command used from both the machines (slow and fast producer):*
>
>kafka-producer-perf-test.sh --broker-list
>x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092
>--messages
>1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats
>--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms
>10000 --compression-codec 2 --reporting-interval 1000
>
>Regards,
>
>Prabhjot
>
>On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik
><Er...@cmegroup.com>
>wrote:
>
>> What happens when the new producer that is getting 70 MB/s is started
>>on a
>> machine that is not part of the kafka cluster?
>>
>> Can you include your topic description/configuration, producer
>> configuration, and broker configuration?
>>
>> On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <pr...@gmail.com> wrote:
>>
>> >Hi,
>> >
>> >I would like to dig deep into this issue. I've changed log4j.properties
>> >for
>> >logging in ALL mode in all places. I am getting lost in the logs.
>> >
>> >Any pointers would be welcome
>> >
>> >Please let me know if you would need any information regarding this
>> >
>> >Thanks,
>> >Prabhjot
>> >
>> >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj
>><pr...@gmail.com>
>> >wrote:
>> >
>> >> Hello Folks,
>> >>
>> >> I've noticed that 2 producer machines, that I had configured, have
>> >>become
>> >> very slow over time
>> >> They are giving 17-19 MB/s
>> >>
>> >> But, a producer that I setup today is giving 70MB/s as the write
>> >>throughput
>> >>
>> >> If I see the contents of bin, libs, config directories, nothing is
>> >> different in the files on any of the producer machines.
>> >>
>> >> Producer is running on the kafka machines itself
>> >>
>> >> Request your expertise
>> >>
>> >> Regards,
>> >> Prabhjot
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >---------------------------------------------------------
>> >"There are only 10 types of people in the world: Those who understand
>> >binary, and those who don't"
>>
>>
>
>
>--
>---------------------------------------------------------
>"There are only 10 types of people in the world: Those who understand
>binary, and those who don't"
Re: Producer becomes slow over time
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi Erik,
I've not tested it on a producer that is not a part of the kafka cluster.
*Producer configuration on all machines:-*
root@x.x.x.x:/a/kafka/config# cat producer.properties | egrep -v '^#|^$'
metadata.broker.list=localhost:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
*Server configuration on all machines:-*
root@y.y.y.y:/a/kafka/config# cat server.properties | egrep -v '^#|^$'
broker.id=0
port=9092
num.network.threads=6
num.io.threads=8
socket.send.buffer.bytes=10485760
socket.receive.buffer.bytes=10485760
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
num.replica.fetchers=4
*Command used from both the machines (slow and fast producer):*
kafka-producer-perf-test.sh --broker-list
x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092 --messages
1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats
--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms
10000 --compression-codec 2 --reporting-interval 1000
Regards,
Prabhjot
On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik <Er...@cmegroup.com>
wrote:
> What happens when the new producer that is getting 70 MB/s is started on a
> machine that is not part of the kafka cluster?
>
> Can you include your topic description/configuration, producer
> configuration, and broker configuration?
>
> On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <pr...@gmail.com> wrote:
>
> >Hi,
> >
> >I would like to dig deep into this issue. I've changed log4j.properties
> >for
> >logging in ALL mode in all places. I am getting lost in the logs.
> >
> >Any pointers would be welcome
> >
> >Please let me know if you would need any information regarding this
> >
> >Thanks,
> >Prabhjot
> >
> >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <pr...@gmail.com>
> >wrote:
> >
> >> Hello Folks,
> >>
> >> I've noticed that 2 producer machines, that I had configured, have
> >>become
> >> very slow over time
> >> They are giving 17-19 MB/s
> >>
> >> But, a producer that I setup today is giving 70MB/s as the write
> >>throughput
> >>
> >> If I see the contents of bin, libs, config directories, nothing is
> >> different in the files on any of the producer machines.
> >>
> >> Producer is running on the kafka machines itself
> >>
> >> Request your expertise
> >>
> >> Regards,
> >> Prabhjot
> >>
> >>
> >>
> >
> >
> >--
> >---------------------------------------------------------
> >"There are only 10 types of people in the world: Those who understand
> >binary, and those who don't"
>
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Producer becomes slow over time
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi Erik,
I've not tested it on a producer that is not a part of the kafka cluster.
*Producer configuration on all machines:-*
root@x.x.x.x:/a/kafka/config# cat producer.properties | egrep -v '^#|^$'
metadata.broker.list=localhost:9092
producer.type=sync
compression.codec=none
serializer.class=kafka.serializer.DefaultEncoder
*Server configuration on all machines:-*
root@y.y.y.y:/a/kafka/config# cat server.properties | egrep -v '^#|^$'
broker.id=0
port=9092
num.network.threads=6
num.io.threads=8
socket.send.buffer.bytes=10485760
socket.receive.buffer.bytes=10485760
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
num.replica.fetchers=4
*Command used from both the machines (slow and fast producer):*
kafka-producer-perf-test.sh --broker-list
x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092 --messages
1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats
--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms
10000 --compression-codec 2 --reporting-interval 1000
Regards,
Prabhjot
On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik <Er...@cmegroup.com>
wrote:
> What happens when the new producer that is getting 70 MB/s is started on a
> machine that is not part of the kafka cluster?
>
> Can you include your topic description/configuration, producer
> configuration, and broker configuration?
>
> On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <pr...@gmail.com> wrote:
>
> >Hi,
> >
> >I would like to dig deep into this issue. I've changed log4j.properties
> >for
> >logging in ALL mode in all places. I am getting lost in the logs.
> >
> >Any pointers would be welcome
> >
> >Please let me know if you would need any information regarding this
> >
> >Thanks,
> >Prabhjot
> >
> >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <pr...@gmail.com>
> >wrote:
> >
> >> Hello Folks,
> >>
> >> I've noticed that 2 producer machines, that I had configured, have
> >>become
> >> very slow over time
> >> They are giving 17-19 MB/s
> >>
> >> But, a producer that I setup today is giving 70MB/s as the write
> >>throughput
> >>
> >> If I see the contents of bin, libs, config directories, nothing is
> >> different in the files on any of the producer machines.
> >>
> >> Producer is running on the kafka machines itself
> >>
> >> Request your expertise
> >>
> >> Regards,
> >> Prabhjot
> >>
> >>
> >>
> >
> >
> >--
> >---------------------------------------------------------
> >"There are only 10 types of people in the world: Those who understand
> >binary, and those who don't"
>
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Producer becomes slow over time
Posted by "Helleren, Erik" <Er...@cmegroup.com>.
What happens when the new producer that is getting 70 MB/s is started on a
machine that is not part of the kafka cluster?
Can you include your topic description/configuration, producer
configuration, and broker configuration?
On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <pr...@gmail.com> wrote:
>Hi,
>
>I would like to dig deep into this issue. I've changed log4j.properties
>for
>logging in ALL mode in all places. I am getting lost in the logs.
>
>Any pointers would be welcome
>
>Please let me know if you would need any information regarding this
>
>Thanks,
>Prabhjot
>
>On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <pr...@gmail.com>
>wrote:
>
>> Hello Folks,
>>
>> I've noticed that 2 producer machines, that I had configured, have
>>become
>> very slow over time
>> They are giving 17-19 MB/s
>>
>> But, a producer that I setup today is giving 70MB/s as the write
>>throughput
>>
>> If I see the contents of bin, libs, config directories, nothing is
>> different in the files on any of the producer machines.
>>
>> Producer is running on the kafka machines itself
>>
>> Request your expertise
>>
>> Regards,
>> Prabhjot
>>
>>
>>
>
>
>--
>---------------------------------------------------------
>"There are only 10 types of people in the world: Those who understand
>binary, and those who don't"
Re: Producer becomes slow over time
Posted by "Helleren, Erik" <Er...@cmegroup.com>.
What happens when the new producer that is getting 70 MB/s is started on a
machine that is not part of the kafka cluster?
Can you include your topic description/configuration, producer
configuration, and broker configuration?
On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <pr...@gmail.com> wrote:
>Hi,
>
>I would like to dig deep into this issue. I've changed log4j.properties
>for
>logging in ALL mode in all places. I am getting lost in the logs.
>
>Any pointers would be welcome
>
>Please let me know if you would need any information regarding this
>
>Thanks,
>Prabhjot
>
>On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <pr...@gmail.com>
>wrote:
>
>> Hello Folks,
>>
>> I've noticed that 2 producer machines, that I had configured, have
>>become
>> very slow over time
>> They are giving 17-19 MB/s
>>
>> But, a producer that I setup today is giving 70MB/s as the write
>>throughput
>>
>> If I see the contents of bin, libs, config directories, nothing is
>> different in the files on any of the producer machines.
>>
>> Producer is running on the kafka machines itself
>>
>> Request your expertise
>>
>> Regards,
>> Prabhjot
>>
>>
>>
>
>
>--
>---------------------------------------------------------
>"There are only 10 types of people in the world: Those who understand
>binary, and those who don't"