You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "eeriee (via GitHub)" <gi...@apache.org> on 2026/08/20 01:10:23 UTC
[PR] Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
eeriee opened a new pull request, #23203:
URL: https://github.com/apache/kafka/pull/23203
RecordAccumulator.ready() runs once per partition, and its per-partition helpers run once per batch, on every Sender runOnce() cycle. This makes the per-partition/per-batch cost matter most for producers with many partitions. This change makes four related improvements:
1. Key batches by TopicPartition instead of partition int
Deques of ProducerBatch were previously keyed by a raw partition int, requiring a TopicPartition -> int mapping step and making it easy to accidentally mix up partitions across topics that share the same partition number. Keying directly by TopicPartition removes that indirection and the associated bookkeeping.
2. Defer Cluster.leaderFor() / MetadataSnapshot.leaderEpochFor() until after the empty-deque check, and short-circuit further per-batch computation when nothing changed
Previously, partitionReady() looked up the leader node and leader epoch for a partition before checking whether its deque had a batch at all. Since the overwhelming majority of partitions have an empty deque on any given cycle (most producers only actively send to a small subset of their partitions at once), this meant paying for a Cluster/TopicPartition-keyed lookup on every partition, every cycle, for no reason. The lookups are now deferred until after confirming the deque actually has a batch, and a checkBatchReady flag lets the subsequent waitedTimeMs/backingOff/full computation and the readyNodes/muted check in batchReady() be skipped entirely when the fast paths already determined the partition isn't ready.
MetadataSnapshot.clusterInstance is now computed eagerly at construction time and stored in a final field (rather than lazily computed and null-checked on every cluster() call), which is safe since a MetadataSnapshot's underlying metadata never changes after construction.
3. Guard trace logging in ProducerBatch.maybeUpdateLeaderEpoch
Both branches called log.trace(...) unconditionally with several varargs, so every call allocated a varargs array and boxed an int even when trace logging was disabled. This method runs once per batch on every ready()/partitionReady() cycle, so at high partition counts the allocation overhead adds up. Guard both calls with log.isTraceEnabled(), matching the existing pattern already used elsewhere in RecordAccumulator/Sender for identically shaped trace logging.
4. Avoid redundant trace logging in BuiltInPartitioner.updatePartitionLoadStats
Only log/reset partitionLoadStatsHolder when actually transitioning out of adaptive mode (i.e. it was previously non-null), instead of unconditionally whenever queueSizes is null. Previously this ran on every ready() cycle for every topic not in adaptive mode, and log.trace()'s internal isTraceEnabled() check can itself be expensive depending on the logging backend's filter chain, so this should be avoided entirely in the (common, steady-state) case where adaptive partitioning was already off on the prior call.
Also adds a JMH benchmark, RecordAccumulatorReadyBenchmark, covering ready() with a large partition count and only a small fraction of partitions having a pending batch (the common case), parameterized on topic count and on whether adaptive partitioning is enabled.
Delete this text and replace it with a detailed description of your change. The
PR title and body will become the squashed commit message.
If you would like to tag individuals, add some commentary, upload images, or
include other supplemental information that should not be part of the eventual
commit message, please use a separate comment.
If applicable, please include a summary of the testing strategy (including
rationale) for the proposed change. Unit and/or integration tests are expected
for any behavior change and system tests should be considered for larger
changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r4096982756
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -320,7 +320,8 @@ public RecordAppendResult append(String topic,
setPartition(callbacks, effectivePartition);
// check if we have an in-progress batch
- Deque<ProducerBatch> dq = topicInfo.batches.computeIfAbsent(effectivePartition, k -> new ArrayDeque<>());
+ TopicPartition tp = new TopicPartition(topic, effectivePartition);
Review Comment:
It used to be that a TopicPartition is created once per batch. With this change, it's created for every record append. Could you do some perf test to make sure that the overall producer send performance doesn't degrade with this change? The included perf test only covers the ready() call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r3893154189
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
@eeriee Have you opened the PR? If not, we can take it over for you if ur bandwidth is limited :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r4243657628
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
Was this followed up on @chia7712 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "eeriee (via GitHub)" <gi...@apache.org>.
eeriee commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r3856087386
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
This has been merged. I'll address this and the nit in another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "eeriee (via GitHub)" <gi...@apache.org>.
eeriee commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r4109303033
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
@chia7712 Sorry i'm too busy these days so i didn't opened a PR. Can you help take it over and also address @junrao's comment? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph merged PR #23203:
URL: https://github.com/apache/kafka/pull/23203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r4268327558
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
This PR is focused on improving the tail latency when the producer is used in the rest-proxy and not using the adaptive partitioning mode. We can revert keying the RecordAccumulator batches by TopicPartition instead of partition int if there is a regression with the regular producer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "eeriee (via GitHub)" <gi...@apache.org>.
eeriee commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r3856087386
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
i'll address this and the nit in another PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r4250848563
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
I have assigned this to a contributor, and he will address Jun's comment as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #23203:
URL: https://github.com/apache/kafka/pull/23203#discussion_r3851419855
##########
jmh-benchmarks/src/main/java/org/apache/kafka/jmh/producer/RecordAccumulatorReadyBenchmark.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.jmh.producer;
+
+import org.apache.kafka.clients.MetadataSnapshot;
+import org.apache.kafka.clients.producer.internals.BufferPool;
+import org.apache.kafka.clients.producer.internals.RecordAccumulator;
+import org.apache.kafka.common.Cluster;
+import org.apache.kafka.common.Node;
+import org.apache.kafka.common.PartitionInfo;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.compress.Compression;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.MetadataResponse.PartitionMetadata;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.internals.LogContext;
+
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Benchmarks {@link RecordAccumulator#ready} (which drives {@code partitionReady()} internally),
+ * the method the Sender background thread calls on every {@code runOnce()} cycle to decide which
+ * nodes have data ready to send. This loop runs once per partition per cycle regardless of
+ * traffic, so its per-partition cost matters most for producers with many partitions.
+ *
+ * <p>Only a small fraction of partitions have a pending batch, matching the common case where
+ * most deques are empty on any given {@code runOnce()} cycle. Adaptive partitioning is
+ * parameterized: when enabled, every partition requires some bookkeeping even with an empty
+ * deque; when disabled, partitions with an empty deque can be skipped almost entirely.
+ *
+ * <p>Partitions are spread across many topics ({@code test-topic-<n>}, 10 partitions each)
+ * instead of one giant topic, so per-partition lookups keyed by {@link TopicPartition} (e.g.
+ * {@code Cluster.leaderFor()}, {@code MetadataSnapshot.leaderEpochFor()}) exercise a realistic
+ * multi-topic key distribution instead of a single-topic one.
+ */
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
+@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+public class RecordAccumulatorReadyBenchmark {
+
+ @Param({"100", "1000"})
+ private int numOfTopics;
+
+ @Param({"false", "true"})
+ private boolean adaptivePartitioning;
+
+ private static final int PARTITIONS_PER_TOPIC = 10;
+
+ private RecordAccumulator accumulator;
+ private MetadataSnapshot metadataSnapshot;
+ private Metrics metrics;
+
+ @Setup(Level.Trial)
+ public void setup() throws InterruptedException {
+ int partitionCount = numOfTopics * PARTITIONS_PER_TOPIC;
+
+ // Only 1% of partitions (at least 1) have a pending batch; the rest have an empty deque,
+ // matching a producer that's only actively sending to a small subset of its partitions
+ // at any moment.
+ int partitionsWithPendingBatch = Math.max(1, partitionCount / 100);
+
+ List<String> topics = new ArrayList<>(numOfTopics);
+ for (int t = 0; t < numOfTopics; t++) {
+ topics.add("test-topic-" + t);
+ }
+
+ Node node = new Node(0, "localhost", 9092);
+ List<Node> nodes = Collections.singletonList(node);
+
+ List<PartitionInfo> partitionInfos = new ArrayList<>(partitionCount);
+ List<PartitionMetadata> partitionMetadatas = new ArrayList<>(partitionCount);
+ for (String topic : topics) {
+ for (int p = 0; p < PARTITIONS_PER_TOPIC; p++) {
+ Node[] replicas = new Node[] {node};
+ partitionInfos.add(new PartitionInfo(topic, p, node, replicas, replicas));
+ List<Integer> replicaIds = Collections.singletonList(node.id());
+ partitionMetadatas.add(new PartitionMetadata(Errors.NONE, new TopicPartition(topic, p),
+ Optional.of(node.id()), Optional.empty(), replicaIds, replicaIds,
+ Collections.emptyList()));
+ }
+ }
+ Cluster cluster = new Cluster("cluster-id", nodes, partitionInfos,
+ Collections.emptySet(), Collections.emptySet());
+
+ Map<Integer, Node> nodesById = new HashMap<>();
+ nodesById.put(node.id(), node);
+ metadataSnapshot = new MetadataSnapshot(null, nodesById, partitionMetadatas,
+ Collections.emptySet(), Collections.emptySet(), Collections.emptySet(), null,
+ Collections.emptyMap(), cluster);
+
+ LogContext logContext = new LogContext();
+ metrics = new Metrics(Time.SYSTEM);
+ int batchSize = 16 * 1024;
+ long totalMemory = (long) (partitionsWithPendingBatch + 16) * batchSize;
+ RecordAccumulator.PartitionerConfig partitionerConfig =
+ new RecordAccumulator.PartitionerConfig(adaptivePartitioning, 0, false, null);
+ accumulator = new RecordAccumulator(
+ logContext,
+ batchSize,
+ Compression.NONE,
+ 0,
+ 100L,
+ 1000L,
+ 120_000,
+ partitionerConfig,
+ metrics,
+ "producer-metrics",
+ Time.SYSTEM,
+ null,
+ new BufferPool(totalMemory, batchSize, metrics, Time.SYSTEM, "producer-metrics"));
+
+ // Only give a subset of partitions a pending batch; the rest keep an empty deque so
+ // ready() takes its early-exit path for them, matching the common case.
+ byte[] key = "key".getBytes(StandardCharsets.UTF_8);
+ byte[] value = new byte[100];
+ long nowMs = Time.SYSTEM.milliseconds();
+ for (int i = 0; i < partitionsWithPendingBatch; i++) {
Review Comment:
What we need to observe is that topicInfo.batches has a bunch of partitions but most of them have an empty queue. This loop only creates entries for the partitions with a non-empty queue, which can't showcase what this PR improves.
##########
clients/src/main/java/org/apache/kafka/clients/MetadataSnapshot.java:
##########
@@ -95,7 +95,7 @@ public MetadataSnapshot(String clusterId,
this.metadataByPartition = Collections.unmodifiableMap(tmpMetadataByPartition);
if (clusterInstance == null) {
- computeClusterView();
+ this.clusterInstance = computeClusterView();
Review Comment:
Nice catch. nit: we could inline `computeClusterView` to make the code clearer :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20917: Reduce per-partition and per-batch overhead in RecordAccumulator.ready() [kafka]
Posted by "eeriee (via GitHub)" <gi...@apache.org>.
eeriee commented on PR #23203:
URL: https://github.com/apache/kafka/pull/23203#issuecomment-5414987136
> @eeriee thanks for this patch. I left two comments for you. Plus, it would be cool if you could show the diff in `alloc/op`, which is stronger evidence :)
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org