0.9.0.1 java consumer reconnection delays

[Igor Velichko <ig...@dataart.com>]

Hello.

While writing integration tests for our solution which uses new Kafka 
consumer group (the one without zookeeper), I faced an issue with 
reconnection. Briefly our test uses an external Kafka server and starts 
a consumer with the predefined group id for each test case, then checks 
that message is received  and afterwards shuts it down. The problem is 
that the consumer not always receives the message despite it works 
correctly.

I figured out that when a consumer disconnects, it could take some time 
for Kafka to remove the consumer from GroupCoordinator and if the 
consumer tries to connect right after it has been closed, the whole 
group restabilization process will be quite slow.  From what I see, the 
time of consumer removal from GroupCoordinator  directly depends on 
session.timeout and the process cannot be forced.

Is there any way to ensure that consumer is fully stopped?

I wrote a small test project to highlight the problem. There consumer is 
started up and tore down in a loop and starts failing after some iterations.
https://github.com/ivelichko/kafka-consumer-reconnect-test

-- 
Best regards,
Igor Velichko