Help required with errors on consumer

[vinay sharma <vi...@gmail.com>]

Hi Everyone,

I am facing some issues with new Kafka consumer API. My consumer polls and
processes records in same thread. I commit explicitly through
commitSync(ConsumerRecord) method. i have seen following issues

1) Sometimes when processing time is too high i see error
"UNKNOWN_MEMBER_ID" on commit and also see "REBALANCE_IN_PROGRESS" error
sometimes.
2) Above problem suggested that probably session is timing out which i
didn't expected as hearbeat task should still be executing time to time.
But to ensure that session doesn't tiomeout I started commiting processed
offsets after 75% time of session time is elapsed and above issue was
resolved. Is this a normal practice with new consumer API?
3) What is significance of consumer config "metadata.max.age.ms"? I see
that if i am processing records after a processing and a metadata request
is issued by NetworkClient then i start getting error "ILLEGAL_GENERATION"
on each commit.
4) What should a slow consumer do in case of a rebalance? if a consumer is
processing and committing and meanwhile a rebalance happen then will this
not cause any issue like "UNKNOWN_MEMBER_ID" or "REBALANCE_IN_PROGRESS"?
what is ideal way to deal with this?

Any pointers will be much appreciated.

Regards,
Vinay Sharma