You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 飞翔的加菲猫 <52...@qq.com> on 2018/09/19 02:58:59 UTC
Questions about manage offset in external storage and consumer failure detect
Hi,
I have two questions about Kafka consumer api:
1. I read the section "Storing Offsets Outside Kafka" in jdoc of KafkaConsumer. Is it safer or more efficient to manage offset with external storage compared to manage with Kafka(Zookeeper or topic _consumer_offset)? Or is there any benefit for system safety or robustness, when network issue occur or consumer crash?
2. I read the section "Detecting Consumer Failures" in jdoc of KafkaConsumder, and got bellow points:
- If consumer not call another poll() after max.poll.interval.ms, it will be remove from consumer group.
- And the only active approach for consumer to detect failures, is to call consumer.commit() and try catch CommitFailedException .
Sometimes I found that consumer hanging at method poll() when network issue occur or rebalance failure, and no exception throw from poll(). How do we detect that failure from consumer side? I refer the old mail "[Kafka-users] How are rebalance failures raised to consumers?" and it seems no approach.
Thanks,
Ruiping Li
Re: Questions about manage offset in external storage and consumer
failure detect
Posted by "Matthias J. Sax" <ma...@confluent.io>.
1. If you don't have a good reason to store offsets externally, I would
not recommend it, but use the client's built-in mechanism. It will be
more work (ie, code you need to write) if you store offsets externally.
For some use-cases, it's beneficial to store the offsets in an external
system to get exaclty-once delivery semantics. But it depends on the use
case if you can apply this pattern.
2.
> "If consumer not call another poll() after max.poll.interval.ms, it will be remove from consumer group."
I think it should be "within" not "after" (can you link to the docs? --
I could not find the snipped you quote)
> And the only active approach for consumer to detect failures, is to call consumer.commit() and try catch CommitFailedException
Not sure what you mean by this? Can you elaborate?
For the blocking poll() issue, this is addressed via
https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior
(available in version 2.0)
-Matthias
On 9/18/18 7:58 PM, 飞翔的加菲猫 wrote:
> Hi,
>
> I have two questions about Kafka consumer api:
>
>
> 1. I read the section "Storing Offsets Outside Kafka" in jdoc of KafkaConsumer. Is it safer or more efficient to manage offset with external storage compared to manage with Kafka(Zookeeper or topic _consumer_offset)? Or is there any benefit for system safety or robustness, when network issue occur or consumer crash?
>
>
> 2. I read the section "Detecting Consumer Failures" in jdoc of KafkaConsumder, and got bellow points:
> - If consumer not call another poll() after max.poll.interval.ms, it will be remove from consumer group.
> - And the only active approach for consumer to detect failures, is to call consumer.commit() and try catch CommitFailedException .
> Sometimes I found that consumer hanging at method poll() when network issue occur or rebalance failure, and no exception throw from poll(). How do we detect that failure from consumer side? I refer the old mail "[Kafka-users] How are rebalance failures raised to consumers?" and it seems no approach.
>
>
> Thanks,
> Ruiping Li
>