Re:Kafka Producer Query - how to increase throughput of producer

[逐风者的祝福 <18...@qq.com>]

hi ! Are you using kafka in akka ?



------------------ Original ------------------
From: "Gnanasoundari Soundarajan"<gn...@in.man.eu>; 
Date: 2018年5月31日(星期四) 晚上11:22
To: "users"<us...@kafka.apache.org>; 
Subject: Kafka Producer Query - how to increase throughput of producer



Hi,

I am new to kafka. I am getting less throughput and high latency in publishing message of size 100-200 bytes.

I have the producer configured with the following configuration. I am using akka-reactive kafka to publish messages
Configuration:
kafka {
        producer {
                parallelism = 500
                bootstrap.servers = "x:9092"
                close-timeout = 86400s
                auto.offset.reset = "earliest"
                max.redeliveries = 0
                use-dispatcher = "kafka-dispatcher"
                kafka-clients {
                retries = 0
                acks = 0
                buffer.memory=67108864
                batch.size=8192
                linger.ms = 2
       }

        }
  }


kafka-dispatcher {
  type = Dispatcher
  executor = "fork-join-executor"
  fork-join-executor {
    parallelism-min = 10
    parallelism-factor = 20.0
    parallelism-max = 300
  }

  throughput = 1
}

Traffic Pattern:
Message published to 10 Topic with 150 partitions.
I am sending the request for 10 Topics with 30 concurrent requests from Jmeter.
I am getting the throughput of 3-5 Hits/sec.

Kafka Server Configuration:

Kafka server and Zookeeper on the same machine centos with 4 CPU 100GB Diskspace and 16 GB RAM.
All are defaults except the below one:

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=50

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=100

Let me know how to increase the throughput of producer ?

Regards,
Gnana