You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Marina <pp...@yahoo.com.INVALID> on 2015/06/08 22:46:26 UTC
what do 'soTimeout', 'bufferSize' and 'minBytes' mean for
SimpleConsumer?
Hi,
I'm using Kafka
0.8.2.1 SimpleConsumer . Could somebody clarify the meaning of a few config parameters for the SimpleConsumer and FetchREquestBuilder? My questions are below:
-- Q1: in the signature of the SimpleConsumer constructor I see the Int 'soTimeout' parameter -
what is the meaning of this timeout? Is this a timeout to connect to the Kafka broker? Timeout on getting a response from any [or specific??] request to Kafka (like FetchRequest)? Something else?
kafka.javaapi.consumer.SimpleConsumer
(val host: String,
val port: Int,
val soTimeout: Int,
val bufferSize: Int,
val clientId: String)
-- Q2: also, SimpleConsumer constructor takes Int 'bufferSize' parameter. What is the meaning
of it? Is this how many bytes SimpleConsumer will read when a fetchRequest is issued?
-- When building FetchREquest via FetchRequestBuilder (see below), I also need to specify 'fetchSize':
FetchRequest req= newFetchRequestBuilder ()
.clientId(kafkaGroupId)
.addFetch(topic, partition, offset, fetchSizeInBytes)
.build();
Looking at the source code of the FetchRequestBuilder , I think (I'm not a Scala pro) those calls
translate into the below method calls - and there the final parameter passed into the FetchRequest is called 'minBytes', hinting that this is not the exact fetch size, possibly….
class FetchRequestBuilder():
def addFetch(topic: String, partition: Int, offset: Long, fetchSize: Int)
def build() = {
val fetchRequest= FetchRequest(versionId, correlationId.getAndIncrement, clientId, replicaId, maxWait, minBytes, requestMap.toMap)
FetchRequest(versionId: Short = FetchRequest.CurrentVersion,
correlationId: Int = FetchRequest.DefaultCorrelationId,
clientId: String = ConsumerConfig.DefaultClientId,
replicaId: Int = Request.OrdinaryConsumerId,
maxWait: Int = FetchRequest.DefaultMaxWait,
minBytes: Int = FetchRequest.DefaultMinBytes,
...)
So, my final question is:
-- Q3: how do 'bufferSize' and 'fetchSize/minBytes' relate? What exactly do they define? Do I have to make
sure one is smaller or grater than the other?
Thanks,
Marina