You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Cliff Rhyne (JIRA)" <ji...@apache.org> on 2015/11/02 20:13:28 UTC
[jira] [Created] (KAFKA-2725) high level consumer rebalances with
auto-commit disabled should throw an exception
Cliff Rhyne created KAFKA-2725:
----------------------------------
Summary: high level consumer rebalances with auto-commit disabled should throw an exception
Key: KAFKA-2725
URL: https://issues.apache.org/jira/browse/KAFKA-2725
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 0.8.2.1
Environment: Experienced on Java running in linux
Reporter: Cliff Rhyne
Auto-commit is a very resilient mode. Drops in zookeeper sessions due to JVM garbage collection, network, rebalance or other interference are handled gracefully within the kafka client.
Systems still can drop due to unexpected gc or network behavior. My proposal is to handle this drop better when auto-commit is turned off:
- If a rebalance or similar occur (which cause the offset to get reverted in the client), check and see if the client was assigned back to the same partition or a different one. If it's the same partition, find the place last consumed (it doesn't do this today for us). This is to make a graceful recovery.
- If the partition assignment changes (which can mean duplicate data is getting processed), throw an exception back to the application code. This lets the application code handle this exception-case with respect to the work it's doing (with might be transactional). Failing "silently" (yes it's still getting logged) is very dangerous in our situation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)