You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by jy...@apache.org on 2026/09/10 13:25:13 UTC
(kafka) branch 4.3 updated (9c58c4260d9 -> 53a7e98898a)
This is an automated email from the ASF dual-hosted git repository.
m1a2st pushed a change to branch 4.3
in repository https://gitbox.apache.org/repos/asf/kafka.git
from 9c58c4260d9 KAFKA-20533: Correcting error response for topic deletion during share fetch (#22170)
add a7b99c40f66 KAFKA-20765: Fix OffsetFetch stale-epoch retry spinning in dedup loop (#22747)
new 27305a9d687 KAFKA-20979: Ensure indexes resize are fsynced when shutting down (#23258)
new 91de2a3b8ce KAFKA-20449: Downgrade "not updating high watermark" log from WARN to DEBUG (#22055)
new 53a7e98898a KAFKA-1983 Remove redundant offset commits from EndToEndLatency (#23402)
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../consumer/internals/CommitRequestManager.java | 31 +++++++----
.../consumer/internals/OffsetFetcherUtils.java | 4 +-
.../internals/CommitRequestManagerTest.java | 60 ++++++++++++++++++++++
.../kafka/storage/internals/log/AbstractIndex.java | 27 +++++++---
.../kafka/storage/internals/log/LogSegment.java | 8 +--
.../storage/internals/log/AbstractIndexTest.java | 17 ++++++
.../org/apache/kafka/tools/EndToEndLatency.java | 25 ++++-----
7 files changed, 135 insertions(+), 37 deletions(-)
(kafka) 02/03: KAFKA-20449: Downgrade "not updating high watermark" log from WARN to DEBUG (#22055)
Posted by jy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
m1a2st pushed a commit to branch 4.3
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 91de2a3b8ce1b7d262537e02697408c3c647047d
Author: Tamas Kornai <ta...@wise.com>
AuthorDate: Tue Apr 14 20:04:45 2026 +0200
KAFKA-20449: Downgrade "not updating high watermark" log from WARN to DEBUG (#22055)
## Summary
- Downgrade the "Not updating high watermark" and "Not updating last
stable offset" log messages in
`OffsetFetcherUtils.updateSubscriptionState()` from `WARN` to `DEBUG`.
- These messages were added in KAFKA-20131 (commit abcbef6a4c) for the
case where a `LIST_OFFSETS` response arrives after a partition has
already been revoked during a rebalance.
- This is a benign race condition that requires no operator action, but
the `WARN` level causes problems for deployments that use warning rates
as a health signal — e.g., canary releases get rolled back due to the
high volume of warnings during rebalances.
## Test plan
- [x] Verify existing `OffsetFetcherUtils` tests pass
- [x] Confirm no tests assert on the log level of these messages
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewers: Lianet Magrans <98...@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <no...@anthropic.com>
---
.../apache/kafka/clients/consumer/internals/OffsetFetcherUtils.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcherUtils.java b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcherUtils.java
index f6f324be070..d2277882277 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcherUtils.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/OffsetFetcherUtils.java
@@ -279,9 +279,9 @@ class OffsetFetcherUtils {
}
} else {
if (isolationLevel == IsolationLevel.READ_COMMITTED) {
- log.warn("Not updating last stable offset for partition {} as it is no longer assigned", partition);
+ log.debug("Not updating last stable offset for partition {} as it is no longer assigned", partition);
} else {
- log.warn("Not updating high watermark for partition {} as it is no longer assigned", partition);
+ log.debug("Not updating high watermark for partition {} as it is no longer assigned", partition);
}
}
}
(kafka) 01/03: KAFKA-20979: Ensure indexes resize are fsynced when shutting down (#23258)
Posted by jy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
m1a2st pushed a commit to branch 4.3
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 27305a9d68769cce22c4e05a15c56078cf80b9aa
Author: Mickael Maison <mi...@users.noreply.github.com>
AuthorDate: Fri Aug 28 12:45:07 2026 +0200
KAFKA-20979: Ensure indexes resize are fsynced when shutting down (#23258)
This avoids reading invalid length for the index files after a clean
shut down
This does not prevent the issue happening on an unclean shutdown. For
validating indexes at startup, we have
https://issues.apache.org/jira/browse/KAFKA-19200
Reviewers: Gaurav Narula <ga...@apple.com>, Luke Chen
<sh...@gmail.com>
---
.../kafka/storage/internals/log/AbstractIndex.java | 27 +++++++++++++++++-----
.../kafka/storage/internals/log/LogSegment.java | 8 +++----
.../storage/internals/log/AbstractIndexTest.java | 17 ++++++++++++++
3 files changed, 42 insertions(+), 10 deletions(-)
diff --git a/storage/src/main/java/org/apache/kafka/storage/internals/log/AbstractIndex.java b/storage/src/main/java/org/apache/kafka/storage/internals/log/AbstractIndex.java
index 7e7c3e1cc63..9545719ea1c 100644
--- a/storage/src/main/java/org/apache/kafka/storage/internals/log/AbstractIndex.java
+++ b/storage/src/main/java/org/apache/kafka/storage/internals/log/AbstractIndex.java
@@ -198,6 +198,15 @@ public abstract class AbstractIndex implements Closeable {
* @return a boolean indicating whether the size of the memory map and the underneath file is changed or not.
*/
public boolean resize(int newSize) throws IOException {
+ return resize(newSize, false);
+ }
+
+ /**
+ * @param newSize new size of the index file
+ * @param sync if true, fsync the file after resizing to ensure the size is durable
+ * @return true if the index was resized, false otherwise
+ */
+ public boolean resize(int newSize, boolean sync) throws IOException {
return inLock(() ->
inRemapWriteLock(() -> {
int roundedNewSize = roundDownToExactMultiple(newSize, entrySize());
@@ -216,6 +225,9 @@ public abstract class AbstractIndex implements Closeable {
mmap = raf.getChannel().map(FileChannel.MapMode.READ_WRITE, 0, roundedNewSize);
this.maxEntries = mmap.limit() / entrySize();
mmap.position(position);
+ if (sync) {
+ raf.getChannel().force(true);
+ }
log.debug("Resized {} to {}, position is {} and limit is {}", file.getAbsolutePath(), roundedNewSize,
mmap.position(), mmap.limit());
return true;
@@ -244,7 +256,9 @@ public abstract class AbstractIndex implements Closeable {
*/
public void flush() {
inLock(() -> {
- mmap.force();
+ if (mmap != null) {
+ mmap.force();
+ }
});
}
@@ -261,13 +275,13 @@ public abstract class AbstractIndex implements Closeable {
}
/**
- * Trim this segment to fit just the valid entries, deleting all trailing unwritten bytes from
- * the file.
+ * Trim this index to fit just the valid entries, deleting all trailing unwritten bytes from the file.
+ * @param sync if true, fsync the file after resizing to ensure both content and size are durable
*/
- public void trimToValidSize() throws IOException {
+ public void trimToValidSize(boolean sync) throws IOException {
inLock(() -> {
if (mmap != null) {
- resize(entrySize() * entries);
+ resize(entrySize() * entries, sync);
}
});
}
@@ -280,7 +294,8 @@ public abstract class AbstractIndex implements Closeable {
}
public void close() throws IOException {
- trimToValidSize();
+ flush(); // Ensure the index content is flushed to disk as LogSegment.close may append an entry
+ trimToValidSize(true);
closeHandler();
}
diff --git a/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java b/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java
index 4d9769e3e10..0b6e2e17f26 100644
--- a/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java
+++ b/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java
@@ -516,10 +516,10 @@ public class LogSegment implements Closeable {
LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
log.truncateTo(validBytes);
- offsetIndex().trimToValidSize();
+ offsetIndex().trimToValidSize(false);
// A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
timeIndex().maybeAppend(maxTimestampSoFar(), shallowOffsetOfMaxTimestampSoFar(), true);
- timeIndex().trimToValidSize();
+ timeIndex().trimToValidSize(false);
return truncated;
}
@@ -680,8 +680,8 @@ public class LogSegment implements Closeable {
*/
public void onBecomeInactiveSegment() throws IOException {
timeIndex().maybeAppend(maxTimestampSoFar(), shallowOffsetOfMaxTimestampSoFar(), true);
- offsetIndex().trimToValidSize();
- timeIndex().trimToValidSize();
+ offsetIndex().trimToValidSize(false);
+ timeIndex().trimToValidSize(false);
log.trim();
}
diff --git a/storage/src/test/java/org/apache/kafka/storage/internals/log/AbstractIndexTest.java b/storage/src/test/java/org/apache/kafka/storage/internals/log/AbstractIndexTest.java
index f6b28345b93..1378577fa95 100644
--- a/storage/src/test/java/org/apache/kafka/storage/internals/log/AbstractIndexTest.java
+++ b/storage/src/test/java/org/apache/kafka/storage/internals/log/AbstractIndexTest.java
@@ -35,11 +35,18 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
public class AbstractIndexTest {
private static class TestIndex extends AbstractIndex {
private boolean unmapInvoked = false;
+ private boolean flushInvoked = false;
private MappedByteBuffer unmappedBuffer = null;
public TestIndex(File file, long baseOffset, int maxIndexSize, boolean writable) throws IOException {
super(file, baseOffset, maxIndexSize, writable);
}
+ @Override
+ public void flush() {
+ flushInvoked = true;
+ super.flush();
+ }
+
@Override
protected int entrySize() {
return 1;
@@ -86,4 +93,14 @@ public class AbstractIndexTest {
assertSame(oldMmap, idx.unmappedBuffer, "old mmap should be unmapped");
assertNotSame(idx.unmappedBuffer, idx.mmap());
}
+
+ @Test
+ public void testCloseInvokeFlush() throws IOException {
+ File f = new File(TestUtils.tempDirectory(), "test-index");
+ TestIndex idx = new TestIndex(f, 0L, 100, true);
+ assertFalse(idx.flushInvoked, "Flush should not have been invoked yet");
+
+ idx.close();
+ assertTrue(idx.flushInvoked, "Flush should have been invoked during close");
+ }
}
(kafka) 03/03: KAFKA-1983 Remove redundant offset commits from EndToEndLatency (#23402)
Posted by jy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
m1a2st pushed a commit to branch 4.3
in repository https://gitbox.apache.org/repos/asf/kafka.git
commit 53a7e98898aec8f8d6a46d2422ae1b0166e5ad7b
Author: Rishabh Saraswat <ri...@nitsri.ac.in>
AuthorDate: Wed Sep 9 02:54:12 2026 +0530
KAFKA-1983 Remove redundant offset commits from EndToEndLatency (#23402)
The tool uses a timestamp-based group.id and calls seekToEnd() on
startup, so committed offsets are never read back. This PR removes the
commitSync() in execute(), replaces commitAndThrow with a plain throw at
its call sites, and deletes the helper.
Reviewers: Chia-Ping Tsai <ch...@gmail.com>
---
.../org/apache/kafka/tools/EndToEndLatency.java | 25 ++++++++--------------
1 file changed, 9 insertions(+), 16 deletions(-)
diff --git a/tools/src/main/java/org/apache/kafka/tools/EndToEndLatency.java b/tools/src/main/java/org/apache/kafka/tools/EndToEndLatency.java
index 486e4f57618..057fbbdc084 100644
--- a/tools/src/main/java/org/apache/kafka/tools/EndToEndLatency.java
+++ b/tools/src/main/java/org/apache/kafka/tools/EndToEndLatency.java
@@ -136,14 +136,13 @@ public class EndToEndLatency {
}
printResults(numRecords, totalTime, latencies);
- consumer.commitSync();
}
}
// Visible for testing
static void validate(KafkaConsumer<byte[], byte[]> consumer, byte[] sentRecordValue, ConsumerRecords<byte[], byte[]> records, byte[] sentRecordKey, Iterable<Header> sentHeaders) {
if (records.isEmpty()) {
- commitAndThrow(consumer, "poll() timed out before finding a result (timeout:[" + POLL_TIMEOUT_MS + "ms])");
+ throw new RuntimeException("poll() timed out before finding a result (timeout:[" + POLL_TIMEOUT_MS + "ms])");
}
ConsumerRecord<byte[], byte[]> record = records.iterator().next();
@@ -151,40 +150,34 @@ public class EndToEndLatency {
String read = new String(record.value(), StandardCharsets.UTF_8);
if (!read.equals(sent)) {
- commitAndThrow(consumer, "The message value read [" + read + "] did not match the message value sent [" + sent + "]");
+ throw new RuntimeException("The message value read [" + read + "] did not match the message value sent [" + sent + "]");
}
if (sentRecordKey != null) {
if (record.key() == null) {
- commitAndThrow(consumer, "Expected message key but received null");
+ throw new RuntimeException("Expected message key but received null");
}
String sentKey = new String(sentRecordKey, StandardCharsets.UTF_8);
String readKey = new String(record.key(), StandardCharsets.UTF_8);
if (!readKey.equals(sentKey)) {
- commitAndThrow(consumer, "The message key read [" + readKey + "] did not match the message key sent [" + sentKey + "]");
+ throw new RuntimeException("The message key read [" + readKey + "] did not match the message key sent [" + sentKey + "]");
}
} else if (record.key() != null) {
- commitAndThrow(consumer, "Expected null message key but received [" + new String(record.key(), StandardCharsets.UTF_8) + "]");
+ throw new RuntimeException("Expected null message key but received [" + new String(record.key(), StandardCharsets.UTF_8) + "]");
}
validateHeaders(consumer, sentHeaders, record);
//Check we only got the one message
if (records.count() != 1) {
- int count = records.count();
- commitAndThrow(consumer, "Only one result was expected during this test. We found [" + count + "]");
+ throw new RuntimeException("Only one result was expected during this test. We found [" + records.count() + "]");
}
}
- private static void commitAndThrow(KafkaConsumer<byte[], byte[]> consumer, String message) {
- consumer.commitSync();
- throw new RuntimeException(message);
- }
-
private static void validateHeaders(KafkaConsumer<byte[], byte[]> consumer, Iterable<Header> sentHeaders, ConsumerRecord<byte[], byte[]> record) {
if (sentHeaders != null && sentHeaders.iterator().hasNext()) {
if (!record.headers().iterator().hasNext()) {
- commitAndThrow(consumer, "Expected message headers but received none");
+ throw new RuntimeException("Expected message headers but received none");
}
Iterator<Header> sentIterator = sentHeaders.iterator();
@@ -196,13 +189,13 @@ public class EndToEndLatency {
if (!receivedHeader.key().equals(sentHeader.key()) || !Arrays.equals(receivedHeader.value(), sentHeader.value())) {
String receivedValueStr = receivedHeader.value() == null ? "null" : Arrays.toString(receivedHeader.value());
String sentValueStr = sentHeader.value() == null ? "null" : Arrays.toString(sentHeader.value());
- commitAndThrow(consumer, "The message header read [" + receivedHeader.key() + ":" + receivedValueStr +
+ throw new RuntimeException("The message header read [" + receivedHeader.key() + ":" + receivedValueStr +
"] did not match the message header sent [" + sentHeader.key() + ":" + sentValueStr + "]");
}
}
if (sentIterator.hasNext() || receivedIterator.hasNext()) {
- commitAndThrow(consumer, "Header count mismatch between sent and received messages");
+ throw new RuntimeException("Header count mismatch between sent and received messages");
}
}
}