You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "squah-confluent (via GitHub)" <gi...@apache.org> on 2024/09/26 13:30:03 UTC

[PR] MINOR: Cache topic resolution in TopicIds set [kafka]

squah-confluent opened a new pull request, #17285:
URL: https://github.com/apache/kafka/pull/17285

   Looking up topics in a TopicsImage is relatively slow. Cache the results
   in TopicIds to improve assignor performance. In benchmarks, we see a
   noticeable improvement in performance in the heterogeneous case.
   
   Before
   ```
   Benchmark                                       (assignmentType)  (assignorType)  (isRackAware)  (memberCount)  (partitionsToMemberRatio)  (subscriptionType)  (topicCount)  Mode  Cnt    Score   Error  Units
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10         HOMOGENEOUS          1000  avgt    5   36.400 ± 3.004  ms/op
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10       HETEROGENEOUS          1000  avgt    5  158.340 ± 0.825  ms/op
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10         HOMOGENEOUS          1000  avgt    5    1.329 ± 0.041  ms/op
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10       HETEROGENEOUS          1000  avgt    5  382.901 ± 6.203  ms/op
   ```
   
   After
   ```
   Benchmark                                       (assignmentType)  (assignorType)  (isRackAware)  (memberCount)  (partitionsToMemberRatio)  (subscriptionType)  (topicCount)  Mode  Cnt    Score   Error  Units
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10         HOMOGENEOUS          1000  avgt    5   36.465 ± 1.954  ms/op
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10       HETEROGENEOUS          1000  avgt    5  114.043 ± 1.424  ms/op
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10         HOMOGENEOUS          1000  avgt    5    1.454 ± 0.019  ms/op
   ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10       HETEROGENEOUS          1000  avgt    5  342.840 ± 2.744  ms/op
   ```
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Cache topic resolution in TopicIds set [kafka]

Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #17285:
URL: https://github.com/apache/kafka/pull/17285#discussion_r1779753418


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -164,20 +308,20 @@ public boolean equals(Object o) {
         TopicIds uuids = (TopicIds) o;
 
         if (!Objects.equals(topicNames, uuids.topicNames)) return false;
-        return Objects.equals(image, uuids.image);
+        return Objects.equals(resolver, uuids.resolver);
     }
 
     @Override
     public int hashCode() {
         int result = topicNames.hashCode();
-        result = 31 * result + image.hashCode();
+        result = 31 * result + resolver.hashCode();
         return result;
     }
 
     @Override
     public String toString() {
         return "TopicIds(topicNames=" + topicNames +
-            ", image=" + image +
+            ", resolver=" + resolver +

Review Comment:
   I'll add a toString implementation that returns `<class name>(image=<image>)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Cache topic resolution in TopicIds set [kafka]

Posted by "squah-confluent (via GitHub)" <gi...@apache.org>.
squah-confluent commented on code in PR #17285:
URL: https://github.com/apache/kafka/pull/17285#discussion_r1779753276


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -21,25 +21,169 @@
 import org.apache.kafka.image.TopicsImage;
 
 import java.util.Collection;
+import java.util.HashMap;
 import java.util.Iterator;
+import java.util.Map;
 import java.util.NoSuchElementException;
 import java.util.Objects;
 import java.util.Set;
 
 /**
  * TopicIds is initialized with topic names (String) but exposes a Set of topic ids (Uuid) to the
- * user and performs the conversion lazily with TopicsImage.
+ * user and performs the conversion lazily with a TopicResolver backed by a TopicsImage.
  */
 public class TopicIds implements Set<Uuid> {
+    /**
+     * Converts between topic ids (Uuids) and topic names (Strings).
+     */
+    public interface TopicResolver {
+        /**
+         * @return The TopicsImage used by the resolver.
+         */
+        TopicsImage image();
+
+        /**
+         * Converts a topic id to a topic name.
+         *
+         * @param id The topic id.
+         * @return The topic name for the given topic id, or null if the topic does not exist.
+         */
+        String name(Uuid id);
+
+        /**
+         * Converts a topic name to a topic id.
+         *
+         * @param name The topic name.
+         * @return The topic id for the given topic name, or null if the topic does not exist.
+         */
+        Uuid id(String name);
+
+        /**
+         * Clears any cached data.
+         *
+         * Used for benchmarking purposes.
+         */
+        void clear();
+    }
+
+    /**
+     * A base implementation of TopicResolver.
+     *
+     * Provides an implementation of equals and hashCode based on the underlying TopicsImage.
+     */
+    public abstract static class BaseTopicResolver implements TopicResolver {

Review Comment:
   Yes, it doesn't need to be public.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR: Cache topic resolution in TopicIds set [kafka]

Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #17285:
URL: https://github.com/apache/kafka/pull/17285#discussion_r1777534739


##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -21,25 +21,169 @@
 import org.apache.kafka.image.TopicsImage;
 
 import java.util.Collection;
+import java.util.HashMap;
 import java.util.Iterator;
+import java.util.Map;
 import java.util.NoSuchElementException;
 import java.util.Objects;
 import java.util.Set;
 
 /**
  * TopicIds is initialized with topic names (String) but exposes a Set of topic ids (Uuid) to the
- * user and performs the conversion lazily with TopicsImage.
+ * user and performs the conversion lazily with a TopicResolver backed by a TopicsImage.
  */
 public class TopicIds implements Set<Uuid> {
+    /**
+     * Converts between topic ids (Uuids) and topic names (Strings).
+     */
+    public interface TopicResolver {
+        /**
+         * @return The TopicsImage used by the resolver.
+         */
+        TopicsImage image();
+
+        /**
+         * Converts a topic id to a topic name.
+         *
+         * @param id The topic id.
+         * @return The topic name for the given topic id, or null if the topic does not exist.
+         */
+        String name(Uuid id);
+
+        /**
+         * Converts a topic name to a topic id.
+         *
+         * @param name The topic name.
+         * @return The topic id for the given topic name, or null if the topic does not exist.
+         */
+        Uuid id(String name);
+
+        /**
+         * Clears any cached data.
+         *
+         * Used for benchmarking purposes.
+         */
+        void clear();
+    }
+
+    /**
+     * A base implementation of TopicResolver.
+     *
+     * Provides an implementation of equals and hashCode based on the underlying TopicsImage.
+     */
+    public abstract static class BaseTopicResolver implements TopicResolver {

Review Comment:
   nit: Should we keep it private?



##########
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/modern/TopicIds.java:
##########
@@ -164,20 +308,20 @@ public boolean equals(Object o) {
         TopicIds uuids = (TopicIds) o;
 
         if (!Objects.equals(topicNames, uuids.topicNames)) return false;
-        return Objects.equals(image, uuids.image);
+        return Objects.equals(resolver, uuids.resolver);
     }
 
     @Override
     public int hashCode() {
         int result = topicNames.hashCode();
-        result = 31 * result + image.hashCode();
+        result = 31 * result + resolver.hashCode();
         return result;
     }
 
     @Override
     public String toString() {
         return "TopicIds(topicNames=" + topicNames +
-            ", image=" + image +
+            ", resolver=" + resolver +

Review Comment:
   nit: If we toString it here, should we add toString methods to the resolvers too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org