You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Ernestas Vaiciukevičius (JIRA)" <ji...@apache.org> on 2017/02/20 15:03:44 UTC
[jira] [Updated] (STORM-2361) Kafka spout - after leader change, it
stops committing offsets to ZK
[ https://issues.apache.org/jira/browse/STORM-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ernestas Vaiciukevičius updated STORM-2361:
-------------------------------------------
Description:
After STORM-2296 although Kafka spouts do not generate duplicates, the offsets committment to ZK may stop on recreated PartitionManagers.
This is because ack's for messages emitted by already destroyed PartitionManagers are not routed properly to the new PartitionManagers handling that partition.
E.g:
``` public void ack(Object msgId) {
KafkaMessageId id = (KafkaMessageId) msgId;
PartitionManager m = _coordinator.getManager(id.partition);
if (m != null) {
m.ack(id.offset);
}
```
id.partition is Partition(host, partition, topic), which is different if Kafka broker changed.
was:
After STORM-2296 although Kafka spouts do not generate duplicates, the offsets committment to ZK may stop on recreated PartitionManagers.
This is because ack's for messages emitted by already destroyed PartitionManagers are not routed properly to the new PartitionManagers handling that partition.
E.g:
` public void ack(Object msgId) {
KafkaMessageId id = (KafkaMessageId) msgId;
PartitionManager m = _coordinator.getManager(id.partition);
if (m != null) {
m.ack(id.offset);
}
`
id.partition is Partition(host, partition, topic), which is different if Kafka broker changed.
> Kafka spout - after leader change, it stops committing offsets to ZK
> --------------------------------------------------------------------
>
> Key: STORM-2361
> URL: https://issues.apache.org/jira/browse/STORM-2361
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
> Affects Versions: 1.1.0
> Reporter: Ernestas Vaiciukevičius
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After STORM-2296 although Kafka spouts do not generate duplicates, the offsets committment to ZK may stop on recreated PartitionManagers.
> This is because ack's for messages emitted by already destroyed PartitionManagers are not routed properly to the new PartitionManagers handling that partition.
> E.g:
> ``` public void ack(Object msgId) {
> KafkaMessageId id = (KafkaMessageId) msgId;
> PartitionManager m = _coordinator.getManager(id.partition);
> if (m != null) {
> m.ack(id.offset);
> }
> ```
> id.partition is Partition(host, partition, topic), which is different if Kafka broker changed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)