You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/08/29 17:37:05 UTC
[GitHub] [kafka] artemlivshits opened a new pull request, #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
artemlivshits opened a new pull request, #12570:
URL: https://github.com/apache/kafka/pull/12570
Now the built-in partitioner defers partition switch (while still
accounting produced bytes) if there is no ready batch to send, thus
avoiding switching partitions and creating fractional batches.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r964385141
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,16 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if there are ready batches in the queue, or we sent all batches.
+ */
+ private boolean queueHasReadyBatches(Deque<ProducerBatch> deque, long nowMs) {
+ // Note that we also check if the queue is empty, because that may mean that batches became
+ // ready and we sent them.
+ ProducerBatch last = deque.peekLast();
+ return deque.size() > 1 || last == null || last.isFull() || last.waitedTimeMs(nowMs) >= lingerMs;
Review Comment:
"Fractional" batches in built-in partitioner could happen with lingerMs=0 as well, and it was probably this consideration that led to the DefaultPartitioner implementation, which switches partitions on batch boundary. The problem with letting the batch fill based on back pressure (which is pretty much the only driver for batching when linger.ms=0) is that we're likely to form larger batches to a slower broker and exhibit the original problem described in KAFKA-10888. This change is carefully avoiding any signals that could be based on backpressure and lets the batch fill only if it's not ready because it doesn't meet readiness criteria itself (full or waited for linger.ms), e.g. we don't let batch fill if there are other ready batches in the queue -- that means the broker wasn't ready to take a ready batch, i.e. backpressure signal.
That said, we could introduce some heuristics, e.g. switch partition if the batch becomes ready and we are within 10% (configurable) of sticky limit, so that instead of starting a new batch in the partition that we switch away from soon, we would start a new batch in the new partition where it may have better opportunity to fill. This would reintroduce some of the KAFKA-10888 pattern, but hopefully the 10% would put a bound on that. The downside of this approach is that it would make the logic even more complex to model and configure, and it's a little research project in itself. I was hoping to get some data from real-life usage of built-in partitioner and see if we need to build the heuristics (as a new project) or the current logic provides good default behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r971047147
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1129,31 +1129,34 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(1, mockRandom.get());
// Produce large record, we should exceed "sticky" limit, but produce to this partition
- // as we switch after the "sticky" limit is exceeded. The partition is switched after
- // we produce.
+ // as we try to switch after the "sticky" limit is exceeded. The switch is disabled
+ // because of incomplete batch.
byte[] largeValue = new byte[batchSize];
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition1, partition.get());
- assertEquals(2, mockRandom.get());
+ assertEquals(1, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
Review Comment:
To be precise, the previous produce didn't switch to the next partition. The produce of this record forces the closing of the current batch, which cause the switch to the next partition.
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1129,31 +1129,34 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(1, mockRandom.get());
// Produce large record, we should exceed "sticky" limit, but produce to this partition
- // as we switch after the "sticky" limit is exceeded. The partition is switched after
- // we produce.
+ // as we try to switch after the "sticky" limit is exceeded. The switch is disabled
+ // because of incomplete batch.
Review Comment:
Thanks for the explanation, Artem. This makes sense to me now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on PR #12570:
URL: https://github.com/apache/kafka/pull/12570#issuecomment-1238270117
@artemlivshits : Thanks for the reply. Could you share some perf results with this PR? It would be useful to test for both 0 and non-zero linger.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on PR #12570:
URL: https://github.com/apache/kafka/pull/12570#issuecomment-1247321505
Looked through the failed tests -- seem unrelated (also ran locally - pass).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r971097010
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1129,31 +1129,34 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(1, mockRandom.get());
// Produce large record, we should exceed "sticky" limit, but produce to this partition
- // as we switch after the "sticky" limit is exceeded. The partition is switched after
- // we produce.
+ // as we try to switch after the "sticky" limit is exceeded. The switch is disabled
+ // because of incomplete batch.
byte[] largeValue = new byte[batchSize];
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition1, partition.get());
- assertEquals(2, mockRandom.get());
+ assertEquals(1, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on PR #12570:
URL: https://github.com/apache/kafka/pull/12570#issuecomment-1235146576
@junrao -- looked through all failed tests, don't seem to use this code path and re-ran them locally, all succeeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r969101389
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java:
##########
@@ -170,13 +170,49 @@ boolean isPartitionChanged(StickyPartitionInfo partitionInfo) {
* @param cluster The cluster information
*/
void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster) {
+ updatePartitionInfo(partitionInfo, appendedBytes, cluster, true);
+ }
+
+ /**
+ * Update partition info with the number of bytes appended and maybe switch partition.
+ * NOTE this function needs to be called under the partition's batch queue lock.
+ *
+ * @param partitionInfo The sticky partition info object returned by peekCurrentPartitionInfo
+ * @param appendedBytes The number of bytes appended to this partition
+ * @param cluster The cluster information
+ * @param enableSwitch If true, switch partition once produced enough bytes
+ */
+ void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster, boolean enableSwitch) {
// partitionInfo may be null if the caller didn't use built-in partitioner.
if (partitionInfo == null)
return;
assert partitionInfo == stickyPartitionInfo.get();
int producedBytes = partitionInfo.producedBytes.addAndGet(appendedBytes);
- if (producedBytes >= stickyBatchSize) {
+
+ // We're trying to switch partition once we produce stickyBatchSize bytes to a partition
+ // but doing so may hinder batching because partition switch may happen while batch isn't
+ // ready to send. This situation is especially likely with high linger.ms setting.
+ // Consider the following example:
+ // linger.ms=500, producer produces 12KB in 500ms, batch.size=16KB
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, then we switch partition, so 4KB gets eventually sent
+ // - ... and so on - we'd get 12KB and 4KB batches
+ // To get more optimal batching and avoid 4KB fractional batches, the caller may disallow
+ // partition switch if batch is not ready to send, so with the example above we'd avoid
+ // fractional 4KB batches: in that case the scenario would look like this:
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, but partition switch doesn't happen because batch in not ready
+ // - second batch collects 12KB in 500ms, gets sent and now we switch partition.
+ // - ... and so on - we'd just send 12KB batches
+ // We cap the produced bytes to not exceed 2x of the batch size to avoid pathological cases
+ // (e.g. if we have a mix of keyed and unkeyed messages, key messages may create an
+ // unready batch after the batch that disabled partition switch becomes ready).
+ // As a result, with high latency.ms setting we end up switching partitions after producing
+ // between stickyBatchSize and stickyBatchSize * 2 bytes, to better align with batch boundary.
+ if (producedBytes >= stickyBatchSize * 2)
+ log.trace("Exceeded {} bytes, produced {} bytes, enable is {}", stickyBatchSize * 2, producedBytes, enableSwitch);
Review Comment:
ok
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1137,23 +1137,26 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(partition1, partition.get());
assertEquals(2, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
+ // for this produce the switch would be disabled because of incomplete batch.
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition2, partition.get());
- assertEquals(3, mockRandom.get());
+ assertEquals(2, mockRandom.get());
Review Comment:
What happens here is the following:
1. First record (small) -- gets a new partition (because there was none)
2. Second record (large) doesn't fit, so the first record forms a batch (but not enough to switch).
3. Second record (large) creates a new batch, but it's not marked as full (disabling the switch).
4. Third record arrives, doesn't fit into the batch, it's marked as full (completing the switch, that was disabled in step 3).
So effectively in the step 4 the switch happens before the records is added, rather than after.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java:
##########
@@ -170,13 +170,49 @@ boolean isPartitionChanged(StickyPartitionInfo partitionInfo) {
* @param cluster The cluster information
*/
void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster) {
+ updatePartitionInfo(partitionInfo, appendedBytes, cluster, true);
+ }
+
+ /**
+ * Update partition info with the number of bytes appended and maybe switch partition.
+ * NOTE this function needs to be called under the partition's batch queue lock.
+ *
+ * @param partitionInfo The sticky partition info object returned by peekCurrentPartitionInfo
+ * @param appendedBytes The number of bytes appended to this partition
+ * @param cluster The cluster information
+ * @param enableSwitch If true, switch partition once produced enough bytes
+ */
+ void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster, boolean enableSwitch) {
// partitionInfo may be null if the caller didn't use built-in partitioner.
if (partitionInfo == null)
return;
assert partitionInfo == stickyPartitionInfo.get();
int producedBytes = partitionInfo.producedBytes.addAndGet(appendedBytes);
- if (producedBytes >= stickyBatchSize) {
+
+ // We're trying to switch partition once we produce stickyBatchSize bytes to a partition
+ // but doing so may hinder batching because partition switch may happen while batch isn't
+ // ready to send. This situation is especially likely with high linger.ms setting.
+ // Consider the following example:
+ // linger.ms=500, producer produces 12KB in 500ms, batch.size=16KB
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, then we switch partition, so 4KB gets eventually sent
+ // - ... and so on - we'd get 12KB and 4KB batches
+ // To get more optimal batching and avoid 4KB fractional batches, the caller may disallow
+ // partition switch if batch is not ready to send, so with the example above we'd avoid
+ // fractional 4KB batches: in that case the scenario would look like this:
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, but partition switch doesn't happen because batch in not ready
+ // - second batch collects 12KB in 500ms, gets sent and now we switch partition.
+ // - ... and so on - we'd just send 12KB batches
+ // We cap the produced bytes to not exceed 2x of the batch size to avoid pathological cases
+ // (e.g. if we have a mix of keyed and unkeyed messages, key messages may create an
+ // unready batch after the batch that disabled partition switch becomes ready).
+ // As a result, with high latency.ms setting we end up switching partitions after producing
+ // between stickyBatchSize and stickyBatchSize * 2 bytes, to better align with batch boundary.
+ if (producedBytes >= stickyBatchSize * 2)
Review Comment:
This could potentially happen if we have a mix of keyed and unkeyed messages.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,15 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if all batches in the queue are full.
+ */
+ private boolean allBatchesFull(Deque<ProducerBatch> deque) {
+ // Only the last batch may be incomplete, so we just check that.
+ ProducerBatch last = deque.peekLast();
+ return last == null || last.isFull();
Review Comment:
Did a bit of experiments and discussed the results offline, looks like the condition as it is would be more backward compatible.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,15 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if all batches in the queue are full.
+ */
+ private boolean allBatchesFull(Deque<ProducerBatch> deque) {
Review Comment:
I think lastBatchFull doesn't include the case of last == null. Alternatively, we could invert the function and name it hasIncompleteBatches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] Kaiserchen commented on pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
Kaiserchen commented on PR #12570:
URL: https://github.com/apache/kafka/pull/12570#issuecomment-1243614928
Just stumbled over this.
Our company basically runs similar code.
Without this optimization MM is non functional for us :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r967463676
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1137,23 +1137,26 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(partition1, partition.get());
assertEquals(2, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
+ // for this produce the switch would be disabled because of incomplete batch.
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition2, partition.get());
- assertEquals(3, mockRandom.get());
+ assertEquals(2, mockRandom.get());
Review Comment:
The partition switching logic is now changed to switch after the batch is full, so just producing the required amount is not enough.
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,15 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if all batches in the queue are full.
+ */
+ private boolean allBatchesFull(Deque<ProducerBatch> deque) {
+ // Only the last batch may be incomplete, so we just check that.
+ ProducerBatch last = deque.peekLast();
+ return last == null || last.isFull();
Review Comment:
That's the only functional change in this commit, the rest is renames, comment updates and unit test fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r967484263
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,15 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if all batches in the queue are full.
+ */
+ private boolean allBatchesFull(Deque<ProducerBatch> deque) {
Review Comment:
allBatchesFull => lastBatchFull ?
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java:
##########
@@ -170,13 +170,49 @@ boolean isPartitionChanged(StickyPartitionInfo partitionInfo) {
* @param cluster The cluster information
*/
void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster) {
+ updatePartitionInfo(partitionInfo, appendedBytes, cluster, true);
+ }
+
+ /**
+ * Update partition info with the number of bytes appended and maybe switch partition.
+ * NOTE this function needs to be called under the partition's batch queue lock.
+ *
+ * @param partitionInfo The sticky partition info object returned by peekCurrentPartitionInfo
+ * @param appendedBytes The number of bytes appended to this partition
+ * @param cluster The cluster information
+ * @param enableSwitch If true, switch partition once produced enough bytes
+ */
+ void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster, boolean enableSwitch) {
// partitionInfo may be null if the caller didn't use built-in partitioner.
if (partitionInfo == null)
return;
assert partitionInfo == stickyPartitionInfo.get();
int producedBytes = partitionInfo.producedBytes.addAndGet(appendedBytes);
- if (producedBytes >= stickyBatchSize) {
+
+ // We're trying to switch partition once we produce stickyBatchSize bytes to a partition
+ // but doing so may hinder batching because partition switch may happen while batch isn't
+ // ready to send. This situation is especially likely with high linger.ms setting.
+ // Consider the following example:
+ // linger.ms=500, producer produces 12KB in 500ms, batch.size=16KB
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, then we switch partition, so 4KB gets eventually sent
+ // - ... and so on - we'd get 12KB and 4KB batches
+ // To get more optimal batching and avoid 4KB fractional batches, the caller may disallow
+ // partition switch if batch is not ready to send, so with the example above we'd avoid
+ // fractional 4KB batches: in that case the scenario would look like this:
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, but partition switch doesn't happen because batch in not ready
+ // - second batch collects 12KB in 500ms, gets sent and now we switch partition.
+ // - ... and so on - we'd just send 12KB batches
+ // We cap the produced bytes to not exceed 2x of the batch size to avoid pathological cases
+ // (e.g. if we have a mix of keyed and unkeyed messages, key messages may create an
+ // unready batch after the batch that disabled partition switch becomes ready).
+ // As a result, with high latency.ms setting we end up switching partitions after producing
+ // between stickyBatchSize and stickyBatchSize * 2 bytes, to better align with batch boundary.
+ if (producedBytes >= stickyBatchSize * 2)
+ log.trace("Exceeded {} bytes, produced {} bytes, enable is {}", stickyBatchSize * 2, producedBytes, enableSwitch);
Review Comment:
Could we rephrase this to be more readable? Sth like "Produced $producedBytes bytes, exceeding twice the batch size of $stickyBatchSize, with switching set to $enableSwitch".
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java:
##########
@@ -170,13 +170,49 @@ boolean isPartitionChanged(StickyPartitionInfo partitionInfo) {
* @param cluster The cluster information
*/
void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster) {
+ updatePartitionInfo(partitionInfo, appendedBytes, cluster, true);
+ }
+
+ /**
+ * Update partition info with the number of bytes appended and maybe switch partition.
+ * NOTE this function needs to be called under the partition's batch queue lock.
+ *
+ * @param partitionInfo The sticky partition info object returned by peekCurrentPartitionInfo
+ * @param appendedBytes The number of bytes appended to this partition
+ * @param cluster The cluster information
+ * @param enableSwitch If true, switch partition once produced enough bytes
+ */
+ void updatePartitionInfo(StickyPartitionInfo partitionInfo, int appendedBytes, Cluster cluster, boolean enableSwitch) {
// partitionInfo may be null if the caller didn't use built-in partitioner.
if (partitionInfo == null)
return;
assert partitionInfo == stickyPartitionInfo.get();
int producedBytes = partitionInfo.producedBytes.addAndGet(appendedBytes);
- if (producedBytes >= stickyBatchSize) {
+
+ // We're trying to switch partition once we produce stickyBatchSize bytes to a partition
+ // but doing so may hinder batching because partition switch may happen while batch isn't
+ // ready to send. This situation is especially likely with high linger.ms setting.
+ // Consider the following example:
+ // linger.ms=500, producer produces 12KB in 500ms, batch.size=16KB
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, then we switch partition, so 4KB gets eventually sent
+ // - ... and so on - we'd get 12KB and 4KB batches
+ // To get more optimal batching and avoid 4KB fractional batches, the caller may disallow
+ // partition switch if batch is not ready to send, so with the example above we'd avoid
+ // fractional 4KB batches: in that case the scenario would look like this:
+ // - first batch collects 12KB in 500ms, gets sent
+ // - second batch collects 4KB, but partition switch doesn't happen because batch in not ready
+ // - second batch collects 12KB in 500ms, gets sent and now we switch partition.
+ // - ... and so on - we'd just send 12KB batches
+ // We cap the produced bytes to not exceed 2x of the batch size to avoid pathological cases
+ // (e.g. if we have a mix of keyed and unkeyed messages, key messages may create an
+ // unready batch after the batch that disabled partition switch becomes ready).
+ // As a result, with high latency.ms setting we end up switching partitions after producing
+ // between stickyBatchSize and stickyBatchSize * 2 bytes, to better align with batch boundary.
+ if (producedBytes >= stickyBatchSize * 2)
Review Comment:
Is this unexpected? If so, should we log as warn?
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1137,23 +1137,26 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(partition1, partition.get());
assertEquals(2, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
+ // for this produce the switch would be disabled because of incomplete batch.
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition2, partition.get());
- assertEquals(3, mockRandom.get());
+ assertEquals(2, mockRandom.get());
Review Comment:
Hmm, in this case, after the append, the last batch is full, which enables switching. Why isn't the partition switched?
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,15 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if all batches in the queue are full.
+ */
+ private boolean allBatchesFull(Deque<ProducerBatch> deque) {
+ // Only the last batch may be incomplete, so we just check that.
+ ProducerBatch last = deque.peekLast();
+ return last == null || last.isFull();
Review Comment:
Would it be better to add the `deque.size() > 1` condition back? Intuitively, this indicates back pressure and we should be allowed to switch away from this partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r970291640
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1129,31 +1129,34 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(1, mockRandom.get());
// Produce large record, we should exceed "sticky" limit, but produce to this partition
- // as we switch after the "sticky" limit is exceeded. The partition is switched after
- // we produce.
+ // as we try to switch after the "sticky" limit is exceeded. The switch is disabled
+ // because of incomplete batch.
Review Comment:
Right now we check the isFull condition, which seems to be false if the batch is not closed and there is one record in the batch -- when a new batch is allocated it is allocated to accommodate the new record, so it can be larger than batch.size (if the first record is greater than batch.size) making the isFull check false. Here is the relevant code
Allocation:
```
int size = Math.max(this.batchSize, AbstractRecords.estimateSizeInBytesUpperBound(maxUsableMagic, compression, key, value, headers));
log.trace("Allocating a new {} byte message buffer for topic {} partition {} with remaining timeout {}ms", size, topic, partition, maxTimeToBlock);
// This call may block if we exhausted buffer space.
buffer = free.allocate(size, maxTimeToBlock);
```
Check:
```
public boolean isFull() {
// note that the write limit is respected only after the first record is added which ensures we can always
// create non-empty batches (this is used to disable batching when the producer's batch size is set to 0).
return appendStream == CLOSED_STREAM || (this.numRecords > 0 && this.writeLimit <= estimatedBytesWritten());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r970263402
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1129,31 +1129,34 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(1, mockRandom.get());
// Produce large record, we should exceed "sticky" limit, but produce to this partition
- // as we switch after the "sticky" limit is exceeded. The partition is switched after
- // we produce.
+ // as we try to switch after the "sticky" limit is exceeded. The switch is disabled
+ // because of incomplete batch.
Review Comment:
I am trying to understand why the switch is disabled here. It seems that the large record won't fit in the current batch. So, we will put the large record in a new batch. This batch will then be full since the large record has more than batch.size bytes in it. This should allow the switch, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on PR #12570:
URL: https://github.com/apache/kafka/pull/12570#issuecomment-1243846932
Thanks for the feedback @Kaiserchen . Could you run your MM test by changing `last == null || last.isFull()` to `deque.size() > 1 || last == null || last.isFull()` and see if it's still functioning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r964182091
##########
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java:
##########
@@ -378,6 +415,16 @@ private MemoryRecordsBuilder recordsBuilder(ByteBuffer buffer, byte maxUsableMag
}
/**
+ * Check if there are ready batches in the queue, or we sent all batches.
+ */
+ private boolean queueHasReadyBatches(Deque<ProducerBatch> deque, long nowMs) {
+ // Note that we also check if the queue is empty, because that may mean that batches became
+ // ready and we sent them.
+ ProducerBatch last = deque.peekLast();
+ return deque.size() > 1 || last == null || last.isFull() || last.waitedTimeMs(nowMs) >= lingerMs;
Review Comment:
Here, we are specifically handling the case with lingerMs > 0. I am wondering if the same issue described in the jira could occur with lingerMs equals to 0. With lingerMs = 0, because of back pressure, the effective batch size is typically between 1 and batch.size. With the built-in partitioner, we could still have the issue with a large batch size followed by a small one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r970071437
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1137,23 +1137,26 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(partition1, partition.get());
assertEquals(2, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
+ // for this produce the switch would be disabled because of incomplete batch.
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition2, partition.get());
- assertEquals(3, mockRandom.get());
+ assertEquals(2, mockRandom.get());
Review Comment:
Thanks, Artem. Sorry, I still don't fully understand.
After step 3, it seems that we switched to partition 2 after the append since mockRandom is 2, right? That part makes sense to me.
In step 4, we append to a new batch in partition 2. After the append(), it seems that enableSwitch should be true since `last.isFull()` should be true. Then, in `topicInfo.builtInPartitioner.updatePartitionInfo`, `producedBytes >= stickyBatchSize && enableSwitch` should be true, which will trigger partition switching. I am wondering what's missing here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r970145425
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1137,23 +1137,26 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(partition1, partition.get());
assertEquals(2, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
+ // for this produce the switch would be disabled because of incomplete batch.
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition2, partition.get());
- assertEquals(3, mockRandom.get());
+ assertEquals(2, mockRandom.get());
Review Comment:
Switching the partition only when the batch is complete make it effectively switch before we create a new batch. In the test the step 2 seems to be an outlier, because the large record + small record seems to exceed the batch.size * 2, so the even though the switch is disabled at the end of step 2, it still happens. I've increased the batch.size in the test, now all tests switch partition before creating a batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] artemlivshits commented on a diff in pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
artemlivshits commented on code in PR #12570:
URL: https://github.com/apache/kafka/pull/12570#discussion_r970145425
##########
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java:
##########
@@ -1137,23 +1137,26 @@ public void onCompletion(RecordMetadata metadata, Exception exception) {
assertEquals(partition1, partition.get());
assertEquals(2, mockRandom.get());
- // Produce large record, we should switch to next partition.
+ // Produce large record, we switched to next partition by previous produce, but
+ // for this produce the switch would be disabled because of incomplete batch.
accum.append(topic, RecordMetadata.UNKNOWN_PARTITION, 0L, null, largeValue, Record.EMPTY_HEADERS,
callbacks, maxBlockTimeMs, false, time.milliseconds(), cluster);
assertEquals(partition2, partition.get());
- assertEquals(3, mockRandom.get());
+ assertEquals(2, mockRandom.get());
Review Comment:
Switching the partition only when the batch is complete make it effectively switch before we create a new batch. In the test the step 2 seems to be an outlier, because the large record + small record seems to exceed the batch.size * 2, so the even though the switch is disabled at the end of step 2, it still happens. I've increase the batch.size in the test, now all tests switch partition before creating a batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao merged pull request #12570: KAFKA-14156: Built-in partitioner may create suboptimal batches
Posted by GitBox <gi...@apache.org>.
junrao merged PR #12570:
URL: https://github.com/apache/kafka/pull/12570
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org