timeout value for consumer.poll

[srimugunthan dhandapani <sr...@gmail.com>]

Hi all,

In our implementation, instead of reading from kafka's automatic offsets
 everytime, we rely on the offsetsForTimes to get the offset for our
interested timestamp and seek to it before we do our consumer.poll()

For quite some time, we could comfortably use 100ms for consumer.poll
timeout. But recently we observe that suddely we require higher timeout
values.
and we have to poll 5 or 6 times with consumer.poll(1000) for the API to
return data.

My questions:
1. how to know what is the ideal timeout value that can be given to the
consumer.poll API?
2. what can cause the sudden spike in the timeouts required ? (If the
ingestion rates are too high in kafka, does that affect the required
consumer poll timeout configuration?)
3. when we have already seeked to the offset where we are sure the data is
present, shouldnt the consumer.poll API return data immediately. Why is
that we are required to poll 5 or 6 times before we get data?

thanks in advance for reply,

-mugunthan