Kafka Spout - offset out of range/offset commit

[Teddy Thomas <TT...@walmartlabs.com>]

Hi all,

First of all thank you for providing such a great tool.

I have a question/problem and I would like to figure out exactly what is happening and workaround/fix it.
A bit of context:
- I have a Kafka Spout (the new one from storm-kafka-client) consuming a Kafka 0.10 topic with 100 partitions
- It may happen that a Kafka consumer gets an offset that is out of range. In that case, the log message “Fetch offset <offset> is out of range for partition <topic-partiton>, resetting offset”
- Once this happen, the partition never gets consumed again and it seems like the new offset (either the latest or earliest depending on the consumer’s reset strategy) never gets committed.
- Re-submitting the topology does Not make the partition be consumed again

I am wondering if it is normal for the new offset to never get committed and the partition to never be consumed again. If yes, what would be the best workaround in order for the partition to get consumed again? It looks like enabling the auto-commit on the Kafka consumer helps, but that lose the reliability that Storm provides :-\

Thank you,

Teddy