You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "squah-confluent (via GitHub)" <gi...@apache.org> on 2024/09/26 13:30:03 UTC
[PR] MINOR: Cache topic resolution in TopicIds set [kafka]
squah-confluent opened a new pull request, #17285:
URL: https://github.com/apache/kafka/pull/17285
Looking up topics in a TopicsImage is relatively slow. Cache the results
in TopicIds to improve assignor performance. In benchmarks, we see a
noticeable improvement in performance in the heterogeneous case.
Before
```
Benchmark (assignmentType) (assignorType) (isRackAware) (memberCount) (partitionsToMemberRatio) (subscriptionType) (topicCount) Mode Cnt Score Error Units
ServerSideAssignorBenchmark.doAssignment INCREMENTAL RANGE false 10000 10 HOMOGENEOUS 1000 avgt 5 36.400 ± 3.004 ms/op
ServerSideAssignorBenchmark.doAssignment INCREMENTAL RANGE false 10000 10 HETEROGENEOUS 1000 avgt 5 158.340 ± 0.825 ms/op
ServerSideAssignorBenchmark.doAssignment INCREMENTAL UNIFORM false 10000 10 HOMOGENEOUS 1000 avgt 5 1.329 ± 0.041 ms/op
ServerSideAssignorBenchmark.doAssignment INCREMENTAL UNIFORM false 10000 10 HETEROGENEOUS 1000 avgt 5 382.901 ± 6.203 ms/op
```
After
```
Benchmark (assignmentType) (assignorType) (isRackAware) (memberCount) (partitionsToMemberRatio) (subscriptionType) (topicCount) Mode Cnt Score Error Units
ServerSideAssignorBenchmark.doAssignment INCREMENTAL RANGE false 10000 10 HOMOGENEOUS 1000 avgt 5 36.465 ± 1.954 ms/op
ServerSideAssignorBenchmark.doAssignment INCREMENTAL RANGE false 10000 10 HETEROGENEOUS 1000 avgt 5 114.043 ± 1.424 ms/op
ServerSideAssignorBenchmark.doAssignment INCREMENTAL UNIFORM false 10000 10 HOMOGENEOUS 1000 avgt 5 1.454 ± 0.019 ms/op
ServerSideAssignorBenchmark.doAssignment INCREMENTAL UNIFORM false 10000 10 HETEROGENEOUS 1000 avgt 5 342.840 ± 2.744 ms/op
```
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Cache topic resolution in TopicIds set [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #17285:
URL: https://github.com/apache/kafka/pull/17285#discussion_r1779753418
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -164,20 +308,20 @@ public boolean equals(Object o) {
TopicIds uuids = (TopicIds) o;
if (!Objects.equals(topicNames, uuids.topicNames)) return false;
- return Objects.equals(image, uuids.image);
+ return Objects.equals(resolver, uuids.resolver);
}
@Override
public int hashCode() {
int result = topicNames.hashCode();
- result = 31 * result + image.hashCode();
+ result = 31 * result + resolver.hashCode();
return result;
}
@Override
public String toString() {
return "TopicIds(topicNames=" + topicNames +
- ", image=" + image +
+ ", resolver=" + resolver +
Review Comment:
I'll add a toString implementation that returns `<class name>(image=<image>)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Cache topic resolution in TopicIds set [kafka]
Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #17285:
URL: https://github.com/apache/kafka/pull/17285#discussion_r1779753276
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -21,25 +21,169 @@
import org.apache.kafka.image.TopicsImage;
import java.util.Collection;
+import java.util.HashMap;
import java.util.Iterator;
+import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Objects;
import java.util.Set;
/**
* TopicIds is initialized with topic names (String) but exposes a Set of topic ids (Uuid) to the
- * user and performs the conversion lazily with TopicsImage.
+ * user and performs the conversion lazily with a TopicResolver backed by a TopicsImage.
*/
public class TopicIds implements Set<Uuid> {
+ /**
+ * Converts between topic ids (Uuids) and topic names (Strings).
+ */
+ public interface TopicResolver {
+ /**
+ * @return The TopicsImage used by the resolver.
+ */
+ TopicsImage image();
+
+ /**
+ * Converts a topic id to a topic name.
+ *
+ * @param id The topic id.
+ * @return The topic name for the given topic id, or null if the topic does not exist.
+ */
+ String name(Uuid id);
+
+ /**
+ * Converts a topic name to a topic id.
+ *
+ * @param name The topic name.
+ * @return The topic id for the given topic name, or null if the topic does not exist.
+ */
+ Uuid id(String name);
+
+ /**
+ * Clears any cached data.
+ *
+ * Used for benchmarking purposes.
+ */
+ void clear();
+ }
+
+ /**
+ * A base implementation of TopicResolver.
+ *
+ * Provides an implementation of equals and hashCode based on the underlying TopicsImage.
+ */
+ public abstract static class BaseTopicResolver implements TopicResolver {
Review Comment:
Yes, it doesn't need to be public.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Cache topic resolution in TopicIds set [kafka]
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #17285:
URL: https://github.com/apache/kafka/pull/17285#discussion_r1777534739
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -21,25 +21,169 @@
import org.apache.kafka.image.TopicsImage;
import java.util.Collection;
+import java.util.HashMap;
import java.util.Iterator;
+import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Objects;
import java.util.Set;
/**
* TopicIds is initialized with topic names (String) but exposes a Set of topic ids (Uuid) to the
- * user and performs the conversion lazily with TopicsImage.
+ * user and performs the conversion lazily with a TopicResolver backed by a TopicsImage.
*/
public class TopicIds implements Set<Uuid> {
+ /**
+ * Converts between topic ids (Uuids) and topic names (Strings).
+ */
+ public interface TopicResolver {
+ /**
+ * @return The TopicsImage used by the resolver.
+ */
+ TopicsImage image();
+
+ /**
+ * Converts a topic id to a topic name.
+ *
+ * @param id The topic id.
+ * @return The topic name for the given topic id, or null if the topic does not exist.
+ */
+ String name(Uuid id);
+
+ /**
+ * Converts a topic name to a topic id.
+ *
+ * @param name The topic name.
+ * @return The topic id for the given topic name, or null if the topic does not exist.
+ */
+ Uuid id(String name);
+
+ /**
+ * Clears any cached data.
+ *
+ * Used for benchmarking purposes.
+ */
+ void clear();
+ }
+
+ /**
+ * A base implementation of TopicResolver.
+ *
+ * Provides an implementation of equals and hashCode based on the underlying TopicsImage.
+ */
+ public abstract static class BaseTopicResolver implements TopicResolver {
Review Comment:
nit: Should we keep it private?
##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -164,20 +308,20 @@ public boolean equals(Object o) {
TopicIds uuids = (TopicIds) o;
if (!Objects.equals(topicNames, uuids.topicNames)) return false;
- return Objects.equals(image, uuids.image);
+ return Objects.equals(resolver, uuids.resolver);
}
@Override
public int hashCode() {
int result = topicNames.hashCode();
- result = 31 * result + image.hashCode();
+ result = 31 * result + resolver.hashCode();
return result;
}
@Override
public String toString() {
return "TopicIds(topicNames=" + topicNames +
- ", image=" + image +
+ ", resolver=" + resolver +
Review Comment:
nit: If we toString it here, should we add toString methods to the resolvers too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org