Kafka Consumer poll

[sat <sa...@gmail.com>]

Hi,

I am new to Kafka. We are planning to use Kafka messaging for our
application. I was playing with Kafka 0.9.0.1 version and i have following
queries. Sorry for asking basic questions.


1) I have instantiated Kafka Consumer and invoked
consumer.poll(Long.MAX_VALUE). Although i have specified timeout as
Long.MAX_VALUE, i observe my consumer to fetch records whenever the
publisher publishes a message to a topic. This makes me wonder whether
Kafka Consumer is push or pull mechanism. Please help us understand the
logic of consumer.poll(timeout).

2) What are the pros and cons of poll for long timeout vs short timeout.


Thanks and Regards
A.SathishKumar

Re: Kafka Consumer poll

[Oleg Zhurakousky <oz...@hortonworks.com>]

Also keep in mind that unfortunately KafkaConsumer.poll(..) will deadlock regardless of the timeout if connection to the broker can not be established and won't react to thread interrupts. This essentially means that the only way to exit is to kill jvm. This is all because Kafka fetches topic metadata synchronously before timeout takes effect.
While it is my understanding that the reason for it is there is a background thread attempting to reconnect in the event of temporary broker outage, it doesn't help if you accidentally specified wrong broker url.

Oleg

> On Aug 2, 2016, at 10:27, Kamal C <ka...@gmail.com> wrote:
> 
> See the answers inline.
> 
>> On Tue, Aug 2, 2016 at 12:23 AM, sat <sa...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> I am new to Kafka. We are planning to use Kafka messaging for our
>> application. I was playing with Kafka 0.9.0.1 version and i have following
>> queries. Sorry for asking basic questions.
>> 
>> 
>> 1) I have instantiated Kafka Consumer and invoked
>> consumer.poll(Long.MAX_VALUE). Although i have specified timeout as
>> Long.MAX_VALUE, i observe my consumer to fetch records whenever the
>> publisher publishes a message to a topic. This makes me wonder whether
>> Kafka Consumer is push or pull mechanism. Please help us understand the
>> logic of consumer.poll(timeout).
> 
> Fetches the data from the topic, waiting up to the specified wait time *if
> necessary *for a record to become available.
> Kafka Consumer by design is pull mechanism.
> 
> Take a look into Kafka Consumer java docs[1]. It's explained in detail.
> 
> 
>> 2) What are the pros and cons of poll for long timeout vs short timeout.
>> 
>> Short Timeout
> 
> Pros:
> - On shutdown, if no data available in the topic -- Shutdown will be quick
> 
> Cons:
> - Number of network trips will be high
> 
> 
>> 
>> Thanks and Regards
>> A.SathishKumar
> 
> [1]:
> https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> 
> -- Kamal

Re: Kafka Consumer poll

[Kamal C <ka...@gmail.com>]

See the answers inline.

On Tue, Aug 2, 2016 at 12:23 AM, sat <sa...@gmail.com> wrote:

> Hi,
>
> I am new to Kafka. We are planning to use Kafka messaging for our
> application. I was playing with Kafka 0.9.0.1 version and i have following
> queries. Sorry for asking basic questions.
>
>
> 1) I have instantiated Kafka Consumer and invoked
> consumer.poll(Long.MAX_VALUE). Although i have specified timeout as
> Long.MAX_VALUE, i observe my consumer to fetch records whenever the
> publisher publishes a message to a topic. This makes me wonder whether
> Kafka Consumer is push or pull mechanism. Please help us understand the
> logic of consumer.poll(timeout).
>

 Fetches the data from the topic, waiting up to the specified wait time *if
necessary *for a record to become available.
 Kafka Consumer by design is pull mechanism.

Take a look into Kafka Consumer java docs[1]. It's explained in detail.


> 2) What are the pros and cons of poll for long timeout vs short timeout.
>
> Short Timeout

Pros:
- On shutdown, if no data available in the topic -- Shutdown will be quick

Cons:
- Number of network trips will be high


>
> Thanks and Regards
> A.SathishKumar
>

[1]:
https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

-- Kamal