You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kirktrue (via GitHub)" <gi...@apache.org> on 2024/04/16 18:56:58 UTC
[PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
kirktrue opened a new pull request, #15737:
URL: https://github.com/apache/kafka/pull/15737
Checking that the `TopicPartition` is in assignment before attempting to remove it.
Also added some logging and refactoring.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1578381540
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,32 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
- def handle_partitions_assigned(self, event):
+ if tp in self.assignment:
+ self.assignment.remove(tp)
+ revoked.append(tp)
+ else:
+ logger.warn("Could not remove topic partition %s from assignment as it was not previously assigned to %s" % (tp, node.account.hostname))
Review Comment:
thanks! better I believe. Do we have a Jira to investigate the failure leading to this? it's concerning (and even more if the case is that is happening with the new protocol only??)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1575365134
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,32 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
- def handle_partitions_assigned(self, event):
+ if tp in self.assignment:
+ self.assignment.remove(tp)
+ revoked.append(tp)
+ else:
+ logger.warn("Could not remove topic partition %s from assignment as it was not previously assigned to %s" % (tp, node.account.hostname))
Review Comment:
@lianetm—I changed the logging to an `assert` that provides useful information for troubleshooting:
```python
tp = _create_partition_from_dict(topic_partition)
assert tp in self.assignment, \
"Topic partition %s cannot be revoked from %s as it was not previously assigned to that consumer" % \
(tp, node.account.hostname)
self.assignment.remove(tp)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1574860975
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,32 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
- def handle_partitions_assigned(self, event):
+ if tp in self.assignment:
+ self.assignment.remove(tp)
+ revoked.append(tp)
+ else:
+ logger.warn("Could not remove topic partition %s from assignment as it was not previously assigned to %s" % (tp, node.account.hostname))
Review Comment:
You’re right @lianetm, this fix could result in a sweeping the problem under the rug, so to speak. I'll change the logic so that this case still results an error, but with more information so we can debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1579722758
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,31 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
+ assert tp in self.assignment, \
+ "Topic partition %s cannot be revoked from %s as it was not previously assigned to that consumer" % \
+ (tp, node.account.hostname)
Review Comment:
@lucasbru—this is the main functional change: ensure that an attempt to remove a partition from the local state verifies that it was previously assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "lianetm (via GitHub)" <gi...@apache.org>.
lianetm commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1569396277
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,32 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
- def handle_partitions_assigned(self, event):
+ if tp in self.assignment:
+ self.assignment.remove(tp)
+ revoked.append(tp)
+ else:
+ logger.warn("Could not remove topic partition %s from assignment as it was not previously assigned to %s" % (tp, node.account.hostname))
Review Comment:
do we understand why this situation is happening? Is it related maybe to the mismatch assignment failure we've seen elsewhere in the tests? My point is just to make sure we're not hiding the real failure with this change. I wouldn't expect that the consumer would ever receive a partition to revoke if it was not previously assigned right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on PR #15737:
URL: https://github.com/apache/kafka/pull/15737#issuecomment-2070963749
@lucasbru—Can you review this change to the consumer system test harness? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1578634103
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,32 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
- def handle_partitions_assigned(self, event):
+ if tp in self.assignment:
+ self.assignment.remove(tp)
+ revoked.append(tp)
+ else:
+ logger.warn("Could not remove topic partition %s from assignment as it was not previously assigned to %s" % (tp, node.account.hostname))
Review Comment:
@lianetm—I will file a JIra on this in the next day or two. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #15737:
URL: https://github.com/apache/kafka/pull/15737#issuecomment-2077011152
Do I understand it correctly that there is no functional change here, just logging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru merged PR #15737:
URL: https://github.com/apache/kafka/pull/15737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-16565: IncrementalAssignmentConsumerEventHandler throws error when attempting to remove a partition that isn't assigned [kafka]
Posted by "kirktrue (via GitHub)" <gi...@apache.org>.
kirktrue commented on code in PR #15737:
URL: https://github.com/apache/kafka/pull/15737#discussion_r1581441497
##########
tests/kafkatest/services/verifiable_consumer.py:
##########
@@ -140,22 +150,32 @@ class IncrementalAssignmentConsumerEventHandler(ConsumerEventHandler):
def __init__(self, node, verify_offsets, idx):
super().__init__(node, verify_offsets, idx)
- def handle_partitions_revoked(self, event):
+ def handle_partitions_revoked(self, event, node, logger):
self.revoked_count += 1
self.state = ConsumerState.Rebalancing
self.position = {}
+ revoked = []
+
for topic_partition in event["partitions"]:
- topic = topic_partition["topic"]
- partition = topic_partition["partition"]
- self.assignment.remove(TopicPartition(topic, partition))
+ tp = _create_partition_from_dict(topic_partition)
- def handle_partitions_assigned(self, event):
+ if tp in self.assignment:
+ self.assignment.remove(tp)
+ revoked.append(tp)
+ else:
+ logger.warn("Could not remove topic partition %s from assignment as it was not previously assigned to %s" % (tp, node.account.hostname))
Review Comment:
Filed KAFKA-16623, FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org