Re: Need to understand the graph

["Wagle, Arun" <aw...@pch.com.INVALID>]

Hi Alexis,

Thanks for the detailed response.

A few questions

 1.  What is the response that is sent back by Kafka. Does it block the client till the message is successfully published/replicated to the servers ? A client here is a REST service which just pushes data to a kafka topic
 2.  Is RequestQueueTime the time required before it is picked up for sending to the processor ?
 3.  What is the ResponseQueueTime and ResponseSendTime. Is this the time after which the data is finally available for consumers ?
 4.  What is RemoteTimeMs ? As per doc it says "Time the request waits for the follower”. Whats a follower in this case.

Thanks in advance

Regards,
Arun Wagle

From: Alexis Midon <al...@airbnb.com>>
Date: Tuesday, March 1, 2016 at 12:30 PM
To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>, Arun Wagle <aw...@pch.com>>
Subject: Re: Need to understand the graph

Also the metrics kafka.network.RequestChannel.RequestQueueSize and ResponseQueueSize will give you the saturation of the network and IO threads.

On Tue, Mar 1, 2016 at 9:21 AM Alexis Midon <al...@airbnb.com>> wrote:
"request queue time" is the time it takes for IO threads to pick up the request. As you increase the load on your broker, it makes sense to see higher queue time.
Here are more details on the request/response model in a Kafka broker (0.8.2).

All your requests and responses are belong to RequestChannel (https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/RequestChannel.scala).

The center piece of all Kafka Request/Response handling is the `kafka.network.RequestChannel`. RequestChannel is a container for all Kafka Requests waiting to be handled, and the Kafka Responses ready to be sent back to the clients.
Requests are queued in a single bounded queue. Up to `queued.max.requests` requests will be stored. The response queues are not bounded.

## First, let's see how the Requests are created.
A SocketServer (created by the main class KafkaServer) starts up multiple threads to handle network connections in a non-blocking fashion:
. a single Acceptor thread
. `network.threads` Processor threads

The Acceptor  thread handles OP_ACCEPT events and passes new socket channel to the Processor threads (with a simple round-robin algorithm).

A Processor thread has 3 responsibilities:
1. register for monitoring the newly created socket channel passed by the Acceptor thread
2. read data from the socket channels until a complete Kafka request is deserialized. The Request is then handed off to the RequestChannel. The hand-off might block if the request queue is full.
3. poll the Kafka Responses queued in the RequestChannel, and write the data on the corresponding socket channel. Each Processor has its own queue in the RequestChannel so that a response is processed by the same thread which read the request from the connection.

## Second, let's see how the Responses are built.
During start up KafkaServer creates an instance of KafkaRequestHandlerPool.  KafkaRequestHandlerPool is a thread pool of `io.thread` threads of KafkaRequestHandler (named "kafka-request-handler-" ).
A KafkaRequestHandler has a fairly simple job: it polls Request from the RequestChannel and passes them to KafkaApis (`KafkaApis.handle()`). KafkaApis dispatches the request and eventually a Response is pushed to the RequestChannel. The response will then be picked up by a Processor thread as described earlier.


A diagram:
http://postimg.org/image/6gadny6px/

The different stages of the request/response handling are nicely measured with the following metrics:
kafka.network.RequestMetrics.RequestQueueTimeMs
kafka.network.RequestMetrics.LocalTimeMs
kafka.network.RequestMetrics.RemoteTimeMs
kafka.network.RequestMetrics.ResponseQueueTimeMs
kafka.network.RequestMetrics.ResponseSendTimeMs
kafka.network.RequestMetrics.TotalTimeMs



On Mon, Feb 29, 2016 at 2:30 PM awagle@pch.com<ma...@pch.com> <st...@gmail.com>> wrote:
Hello,

I have an issue with the timing of Kafka. As and when the time increases with load testing, we see the increase in "request queue time". What is the "request queue time" ?

Some basic config we are using

2 Kafka Nodes(CPU - 8 CPU's [Thread(s) per core:    2], Mem - 32 GB Ram)
pkafkaapp01.pchoso.com<http://pkafkaapp01.pchoso.com>
pkafkaapp02.pchoso.com<http://pkafkaapp02.pchoso.com>

Tests duration: 13:05 - 14:05
Messages published - 869156

Load Average, CPU, memory is all under control so not sure what the issue is.

Below are some SPM graphs showing the state of my system.
Here's the 'Requests' graph:
  https://apps.sematext.com/spm-reports/s/lCOJULIKuJ