You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Marina <pp...@yahoo.com.INVALID> on 2015/06/08 22:46:26 UTC

what do 'soTimeout', 'bufferSize' and 'minBytes' mean for SimpleConsumer?

Hi, 
I'm using Kafka
0.8.2.1 SimpleConsumer . Could somebody clarify the meaning of a few config parameters for the SimpleConsumer and FetchREquestBuilder? My questions are below:
 
-- Q1: in the signature of the SimpleConsumer constructor I see the Int 'soTimeout' parameter -
what is the meaning of this timeout? Is this a timeout to connect to the Kafka broker? Timeout on getting a response from any [or specific??] request to Kafka (like FetchRequest)? Something else?
 
kafka.javaapi.consumer.SimpleConsumer
    (val host: String,
    val port: Int,
    val soTimeout: Int,
    val bufferSize: Int,
    val clientId: String) 
 


-- Q2: also, SimpleConsumer constructor takes Int 'bufferSize' parameter. What is the meaning
of it? Is this how many bytes SimpleConsumer will read when a fetchRequest is issued? 
 
-- When building FetchREquest via FetchRequestBuilder (see below), I also need to specify 'fetchSize': 
 
FetchRequest req= newFetchRequestBuilder ()
    .clientId(kafkaGroupId)
    .addFetch(topic, partition, offset, fetchSizeInBytes)
    .build();
 
Looking at the source code of the FetchRequestBuilder , I think (I'm not a Scala pro) those calls
translate into the below method calls - and there the final parameter passed into the FetchRequest is called 'minBytes', hinting that this is not the exact fetch size, possibly….
 
class FetchRequestBuilder(): 
    def addFetch(topic: String, partition: Int, offset: Long, fetchSize: Int)
 
    def build() = {
    val fetchRequest= FetchRequest(versionId, correlationId.getAndIncrement, clientId, replicaId, maxWait, minBytes, requestMap.toMap)
 
FetchRequest(versionId: Short = FetchRequest.CurrentVersion,
    correlationId: Int = FetchRequest.DefaultCorrelationId,
    clientId: String = ConsumerConfig.DefaultClientId,
    replicaId: Int = Request.OrdinaryConsumerId,
    maxWait: Int = FetchRequest.DefaultMaxWait,
    minBytes: Int = FetchRequest.DefaultMinBytes,
    ...)
 
 
So, my final question is: 
-- Q3: how do 'bufferSize' and 'fetchSize/minBytes' relate? What exactly do they define? Do I have to make
sure one is smaller or grater than the other?
 
Thanks,
Marina