You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Hawin Jiang <ha...@gmail.com> on 2015/07/03 02:58:24 UTC
Kafka bottleneck issue
Hi Admin
I have executed a performance test on one single node.The TPS never over
4000 events per sec on one producer.Please help me.
Thanks in advance.
Here is my code and configuration
*********************Producer code***************************************
public void run() {
int messageNo = 1;
while(true)
{
String messageStr = new String("Message_" + messageNo);
producer.send(new KeyedMessage<Integer, String>(topic, messageStr));
messageNo++;
}
}
************************server.properties***********************************
broker.id=1
port=9092
host.name=192.168.0.112
num.network.threads=8
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
#log.flush.interval.ms=1000
log.retention.hours=168
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=192.168.0.112:2181
zookeeper.connection.timeout.ms=6000
zookeeper.properties
***************************************************************************************
*************************************
zookeeper.properties***************************
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
****************************************************************************************
Best regards
Hawin
Re: Kafka bottleneck issue
Posted by Guozhang Wang <wa...@gmail.com>.
Hawin,
I suggest you to use ProducerPerformance to test the throughput again, just
to see if the possible bottleneck can be excluded from the Kafka side.
Guozhang
On Thu, Jul 2, 2015 at 6:32 PM, Hawin Jiang <ha...@gmail.com> wrote:
> Hi Aditya
>
> Yes. I have a single producer thread. I used example code to test it.
> I started a producer thread and sent messages to producer for one minutes.
> if I start two producer thread. I will receive around 8000 TPS(
> Transactions per second).
> I just don't want to start many threads to receive high TPS.
> Thanks.
>
>
>
> Best regards
> Hawin
>
> On Thu, Jul 2, 2015 at 6:26 PM, Aditya Auradkar <
> aauradkar@linkedin.com.invalid> wrote:
>
> > Do you have a single producer thread? Also, how do you calculate the TPS
> > (is it transactions per second?) ?
> >
> > Aditya
> >
> > ________________________________________
> > From: Hawin Jiang [hawin.jiang@gmail.com]
> > Sent: Thursday, July 02, 2015 5:58 PM
> > To: users@kafka.apache.org
> > Subject: Kafka bottleneck issue
> >
> > Hi Admin
> >
> > I have executed a performance test on one single node.The TPS never over
> > 4000 events per sec on one producer.Please help me.
> > Thanks in advance.
> >
> >
> > Here is my code and configuration
> >
> > *********************Producer code***************************************
> > public void run() {
> > int messageNo = 1;
> > while(true)
> > {
> > String messageStr = new String("Message_" + messageNo);
> > producer.send(new KeyedMessage<Integer, String>(topic,
> messageStr));
> > messageNo++;
> > }
> > }
> >
> >
> >
> >
> ************************server.properties***********************************
> > broker.id=1
> > port=9092
> > host.name=192.168.0.112
> > num.network.threads=8
> > num.io.threads=8
> > socket.send.buffer.bytes=1048576
> > socket.receive.buffer.bytes=1048576
> > socket.request.max.bytes=104857600
> > log.dirs=/tmp/kafka-logs
> > num.partitions=1
> > num.recovery.threads.per.data.dir=1
> > #log.flush.interval.ms=1000
> > log.retention.hours=168
> > #log.retention.bytes=1073741824
> > log.segment.bytes=1073741824
> > log.retention.check.interval.ms=300000
> > log.cleaner.enable=false
> > zookeeper.connect=192.168.0.112:2181
> > zookeeper.connection.timeout.ms=6000
> > zookeeper.properties
> >
> >
> ***************************************************************************************
> >
> >
> > *************************************
> > zookeeper.properties***************************
> > dataDir=/tmp/zookeeper
> > clientPort=2181
> > maxClientCnxns=0
> >
> >
> ****************************************************************************************
> >
> >
> >
> >
> > Best regards
> > Hawin
> >
>
--
-- Guozhang
Re: Kafka bottleneck issue
Posted by Hawin Jiang <ha...@gmail.com>.
Hi Aditya
Yes. I have a single producer thread. I used example code to test it.
I started a producer thread and sent messages to producer for one minutes.
if I start two producer thread. I will receive around 8000 TPS(
Transactions per second).
I just don't want to start many threads to receive high TPS.
Thanks.
Best regards
Hawin
On Thu, Jul 2, 2015 at 6:26 PM, Aditya Auradkar <
aauradkar@linkedin.com.invalid> wrote:
> Do you have a single producer thread? Also, how do you calculate the TPS
> (is it transactions per second?) ?
>
> Aditya
>
> ________________________________________
> From: Hawin Jiang [hawin.jiang@gmail.com]
> Sent: Thursday, July 02, 2015 5:58 PM
> To: users@kafka.apache.org
> Subject: Kafka bottleneck issue
>
> Hi Admin
>
> I have executed a performance test on one single node.The TPS never over
> 4000 events per sec on one producer.Please help me.
> Thanks in advance.
>
>
> Here is my code and configuration
>
> *********************Producer code***************************************
> public void run() {
> int messageNo = 1;
> while(true)
> {
> String messageStr = new String("Message_" + messageNo);
> producer.send(new KeyedMessage<Integer, String>(topic, messageStr));
> messageNo++;
> }
> }
>
>
>
> ************************server.properties***********************************
> broker.id=1
> port=9092
> host.name=192.168.0.112
> num.network.threads=8
> num.io.threads=8
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=1048576
> socket.request.max.bytes=104857600
> log.dirs=/tmp/kafka-logs
> num.partitions=1
> num.recovery.threads.per.data.dir=1
> #log.flush.interval.ms=1000
> log.retention.hours=168
> #log.retention.bytes=1073741824
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=300000
> log.cleaner.enable=false
> zookeeper.connect=192.168.0.112:2181
> zookeeper.connection.timeout.ms=6000
> zookeeper.properties
>
> ***************************************************************************************
>
>
> *************************************
> zookeeper.properties***************************
> dataDir=/tmp/zookeeper
> clientPort=2181
> maxClientCnxns=0
>
> ****************************************************************************************
>
>
>
>
> Best regards
> Hawin
>
RE: Kafka bottleneck issue
Posted by Aditya Auradkar <aa...@linkedin.com.INVALID>.
Do you have a single producer thread? Also, how do you calculate the TPS (is it transactions per second?) ?
Aditya
________________________________________
From: Hawin Jiang [hawin.jiang@gmail.com]
Sent: Thursday, July 02, 2015 5:58 PM
To: users@kafka.apache.org
Subject: Kafka bottleneck issue
Hi Admin
I have executed a performance test on one single node.The TPS never over
4000 events per sec on one producer.Please help me.
Thanks in advance.
Here is my code and configuration
*********************Producer code***************************************
public void run() {
int messageNo = 1;
while(true)
{
String messageStr = new String("Message_" + messageNo);
producer.send(new KeyedMessage<Integer, String>(topic, messageStr));
messageNo++;
}
}
************************server.properties***********************************
broker.id=1
port=9092
host.name=192.168.0.112
num.network.threads=8
num.io.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
#log.flush.interval.ms=1000
log.retention.hours=168
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=192.168.0.112:2181
zookeeper.connection.timeout.ms=6000
zookeeper.properties
***************************************************************************************
*************************************
zookeeper.properties***************************
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
****************************************************************************************
Best regards
Hawin