You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "keemgdeok (via GitHub)" <gi...@apache.org> on 2026/10/14 13:36:43 UTC
[PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
keemgdeok opened a new pull request, #23816:
URL: https://github.com/apache/kafka/pull/23816
Verify the log start offsets and HW/LSO/LEO after the active segment is rolled and offloaded. This covers the case where all produced records are remote and the remaining local segment is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on PR #23816:
URL: https://github.com/apache/kafka/pull/23816#issuecomment-6176253349
The offsets are only checked once, at the end. Running the check right after produce and the offload wait could be better for HW/LEO ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #23816:
URL: https://github.com/apache/kafka/pull/23816#discussion_r4275121859
##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/RollAndOffloadActiveSegmentTest.java:
##########
@@ -89,11 +94,21 @@ private void executeRollAndOffloadActiveSegmentTest(ClusterInstance clusterInsta
.expectEarliestLocalOffsetInLogDirectory(topicA, p0, 4L)
.produce(topicA, p0, new KeyValueSpec("k0", "v0"), new KeyValueSpec("k1", "v1"),
new KeyValueSpec("k2", "v2"), new KeyValueSpec("k3", "v3"))
+ // Local retention must not advance the start offset of data still available remotely.
+ .expectListOffsets(topicA, p0, OffsetSpec.earliest(), new EpochEntry(0, 0))
+ .expectListOffsets(topicA, p0, OffsetSpec.earliestLocal(), new EpochEntry(0, 4))
// consume from the beginning of the topic to read data from local and remote storage
.expectFetchFromTieredStorage(broker0, topicA, p0, 4)
.consume(topicA, p0, 0L, 4, 4);
builder.build().execute(clusterInstance, groupProtocol);
+
+ // Rolling the final segment must not advance offsets beyond the four produced records.
+ UnifiedLog log = clusterInstance.brokers().get(broker0).logManager()
Review Comment:
Looks like these assertions run after the execute has already printed the report. So a failure here comes with no context probably.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #23816:
URL: https://github.com/apache/kafka/pull/23816#discussion_r4275102977
##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/RollAndOffloadActiveSegmentTest.java:
##########
@@ -89,11 +94,21 @@ private void executeRollAndOffloadActiveSegmentTest(ClusterInstance clusterInsta
.expectEarliestLocalOffsetInLogDirectory(topicA, p0, 4L)
.produce(topicA, p0, new KeyValueSpec("k0", "v0"), new KeyValueSpec("k1", "v1"),
new KeyValueSpec("k2", "v2"), new KeyValueSpec("k3", "v3"))
+ // Local retention must not advance the start offset of data still available remotely.
+ .expectListOffsets(topicA, p0, OffsetSpec.earliest(), new EpochEntry(0, 0))
+ .expectListOffsets(topicA, p0, OffsetSpec.earliestLocal(), new EpochEntry(0, 4))
Review Comment:
Can you also pls add latestTiered check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
Posted by "keemgdeok (via GitHub)" <gi...@apache.org>.
keemgdeok commented on code in PR #23816:
URL: https://github.com/apache/kafka/pull/23816#discussion_r4282584703
##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/RollAndOffloadActiveSegmentTest.java:
##########
@@ -89,11 +94,21 @@ private void executeRollAndOffloadActiveSegmentTest(ClusterInstance clusterInsta
.expectEarliestLocalOffsetInLogDirectory(topicA, p0, 4L)
.produce(topicA, p0, new KeyValueSpec("k0", "v0"), new KeyValueSpec("k1", "v1"),
new KeyValueSpec("k2", "v2"), new KeyValueSpec("k3", "v3"))
+ // Local retention must not advance the start offset of data still available remotely.
+ .expectListOffsets(topicA, p0, OffsetSpec.earliest(), new EpochEntry(0, 0))
+ .expectListOffsets(topicA, p0, OffsetSpec.earliestLocal(), new EpochEntry(0, 4))
// consume from the beginning of the topic to read data from local and remote storage
.expectFetchFromTieredStorage(broker0, topicA, p0, 4)
.consume(topicA, p0, 0L, 4, 4);
builder.build().execute(clusterInstance, groupProtocol);
+
+ // Rolling the final segment must not advance offsets beyond the four produced records.
+ UnifiedLog log = clusterInstance.brokers().get(broker0).logManager()
Review Comment:
Moved the checks into `ExpectLogOffsetsAction` after `produce()`, before the explicit consume. The runner now records assertion failures before printing the report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
Posted by "keemgdeok (via GitHub)" <gi...@apache.org>.
keemgdeok commented on code in PR #23816:
URL: https://github.com/apache/kafka/pull/23816#discussion_r4282578264
##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/RollAndOffloadActiveSegmentTest.java:
##########
@@ -89,11 +94,21 @@ private void executeRollAndOffloadActiveSegmentTest(ClusterInstance clusterInsta
.expectEarliestLocalOffsetInLogDirectory(topicA, p0, 4L)
.produce(topicA, p0, new KeyValueSpec("k0", "v0"), new KeyValueSpec("k1", "v1"),
new KeyValueSpec("k2", "v2"), new KeyValueSpec("k3", "v3"))
+ // Local retention must not advance the start offset of data still available remotely.
+ .expectListOffsets(topicA, p0, OffsetSpec.earliest(), new EpochEntry(0, 0))
+ .expectListOffsets(topicA, p0, OffsetSpec.earliestLocal(), new EpochEntry(0, 4))
Review Comment:
Added `latestTiered()` with expected offset 3 in 79ac61b.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-15864: Verify offsets after rolling and offloading the active segment [kafka]
Posted by "keemgdeok (via GitHub)" <gi...@apache.org>.
keemgdeok commented on PR #23816:
URL: https://github.com/apache/kafka/pull/23816#issuecomment-6191365349
Updated in 79ac61b. The offset checks now run as the next action after `produce()`, before the explicit consume.
Java 17/macOS ARM64: both consumer protocols in `RollAndOffloadActiveSegmentTest` and `ListOffsetsTest`, plus the three failure-reporting cases, passed (7 tests). Storage Checkstyle and Spotless passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org