You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "muralibasani (via GitHub)" <gi...@apache.org> on 2026/07/19 13:45:33 UTC
[PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
muralibasani opened a new pull request, #22873:
URL: https://github.com/apache/kafka/pull/22873
Ref : https://issues.apache.org/jira/browse/KAFKA-20609
In tiered storage, a segment with a future record timestamp was never removed from local disk by time-based retention. since its age was computed from that future timestamp — pinning it locally even after it was safely uploaded to remote storage.
In this change - introduced ageReferenceMs variable in deleteRetentionMsBreachedSegments() that normally holds largestTimestamp(), but falls back to the segment's lastModified time when the record timestamp is in the future and remote copy is enabled.
Already-offloaded segments then age out of local storage as expected. The existing upload-before-delete guard (isSegmentEligibleForDeletion) keeps data safe, and non-tiered logs are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #22873:
URL: https://github.com/apache/kafka/pull/22873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4109605690
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
> after it's uploaded to tier storage
Agreed, and that is similar to my previous suggestion "the copy-finished timestamp in remote metadata".
Another point we need to discuss is whether it belongs to the fallback mechanism or we should just ignore the record timestamp. I think we can do the former as a kind of improvement, and the latter probably needs a KIP to change the semantic of local retention. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3937723440
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2005,34 +2005,30 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled()) {
Review Comment:
Yes, refactored it and moved to top as bool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4100427371
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
> occurs even when tiered storage is not used
Yes, we had considered it as well. However, the risk is different from the remote storage case. The segments which have been uploaded can't be deleted locally. That violates the purpose of remote storage. Also, it will block the deletion of the following segments, potentially causing a full disk.
We can have a similar fallback on the normal path, but I did not see a strong reason for it. I can file a JIRA to track it and wait for a good reason. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3731514190
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2004,11 +2004,18 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long ageReferenceMs = segment.largestTimestamp();
Review Comment:
Done, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4124683328
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
If we think the new semantic makes more intuitive sense, it's better to use it consistently regardless of the value of the event time. I agree that we will need a KIP for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4101730943
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
Apologies for jumping ahead on the revert, @chia7712. Thanks both.
I'll keep the lastModified fallback for data that's already stored (limited delay vs. stuck forever), use KAFKA-21158 to stop new cases with the config doc and warning, and switch to a stable anchor in your follow-up JIRA. @junrao, hope this is good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3931550937
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2005,34 +2005,30 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled()) {
Review Comment:
Should we evaluate `remoteLogEnabledAndRemoteCopyEnabled()` once to keep it consistent within this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4098186330
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
Sure. I shall follow up on this https://issues.apache.org/jira/browse/KAFKA-21158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4097159048
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
The problem with lastModified is that it may not be accurate after reassignment or disk replacement.
Have we considered to use [log.message.timestamp.after.max.ms](https://kafka.apache.org/43/configuration/broker-configs/#brokerconfigs_log.message.timestamp.after.max.ms) to prevent future timestamps from leaking in in the first place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on PR #22873:
URL: https://github.com/apache/kafka/pull/22873#issuecomment-5208714673
> @muralibasani thanks for your patch. overall lgtm
@chia7712 thank you for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3937728510
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2005,34 +2005,30 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled()) {
+ anchorTimestamp = segment.lastModified();
+ futureTimestampLogger.warn("{} contains future timestamp(s), using lastModified time {} as the retention anchor", segment, anchorTimestamp);
+ } else {
+ futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ }
}
- boolean delete = startMs - segment.largestTimestamp() > retentionMs;
+ boolean delete = startMs - anchorTimestamp > retentionMs;
logger.debug("{} retentionMs breached: {}, startMs={}, retentionMs={}",
segment, delete, startMs, retentionMs);
return delete;
};
return deleteOldSegments(shouldDelete, toDelete -> {
- long localRetentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long localRetentionMs = UnifiedLog.localRetentionMs(config(), remoteCopyEnabled);
Review Comment:
Yes, that was a duplicate. Dropped it. Log now uses the existing retentionMs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4098029643
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
The retention issue due to future timestamp occurs even when tiered storage is not used. We can probably document this in the config doc and add a warning if log.message.timestamp.after.max.ms is increased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3912215351
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2005,10 +2005,16 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled() && segment.largestRecordTimestamp().isPresent()) {
Review Comment:
It seems `segment.largestRecordTimestamp().isPresent()` is redundant, right? When it's empty, `largestTimestamp()` already returns `lastModified()`, so the result is the same with or without it :)
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
Review Comment:
This log message hasn't heard about the PR yet :)
##########
storage/src/test/java/org/apache/kafka/tiered/storage/integration/DeleteSegmentsTest.java:
##########
@@ -104,19 +107,23 @@ private static void executeDeleteSegmentsTest(ClusterInstance clusterInstance,
.expectSegmentToBeOffloaded(broker0, topicA, p0, 2, new KeyValueSpec("k2", "v2"))
.expectEarliestLocalOffsetInLogDirectory(topicA, p0, 3L)
.produceWithTimestamp(topicA, p0, new KeyValueSpec("k0", "v0"), new KeyValueSpec("k1", "v1"),
- // testDeleteSegmentsByRetentionTime* uses a tiny retention time, which could cause the active
- // segment to be rolled and deleted. We use a future timestamp to prevent that from happening.
Review Comment:
Would you mind adding the new behavior instead of removing previous comments?
##########
storage/src/test/java/org/apache/kafka/tiered/storage/actions/WaitForEarliestLocalOffsetAction.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.tiered.storage.actions;
+
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.tiered.storage.TieredStorageTestAction;
+import org.apache.kafka.tiered.storage.TieredStorageTestContext;
+import org.apache.kafka.tiered.storage.utils.BrokerLocalStorage;
+
+import java.io.PrintStream;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+
+public final class WaitForEarliestLocalOffsetAction implements TieredStorageTestAction {
+
+ private final TopicPartition topicPartition;
+ private final Long earliestLocalOffset;
+
+ public WaitForEarliestLocalOffsetAction(TopicPartition topicPartition,
+ Long earliestLocalOffset) {
Review Comment:
`long`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3931564987
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2005,34 +2005,30 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled()) {
+ anchorTimestamp = segment.lastModified();
+ futureTimestampLogger.warn("{} contains future timestamp(s), using lastModified time {} as the retention anchor", segment, anchorTimestamp);
+ } else {
+ futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ }
}
- boolean delete = startMs - segment.largestTimestamp() > retentionMs;
+ boolean delete = startMs - anchorTimestamp > retentionMs;
logger.debug("{} retentionMs breached: {}, startMs={}, retentionMs={}",
segment, delete, startMs, retentionMs);
return delete;
};
return deleteOldSegments(shouldDelete, toDelete -> {
- long localRetentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long localRetentionMs = UnifiedLog.localRetentionMs(config(), remoteCopyEnabled);
Review Comment:
`localRetentionMs` is duplicate to `retentionMs`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4106463939
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
The problem with using lastModified is that it can still delay the deletion of local segments and it feels a bit hacky.
It would be useful to think through the semantic of local retention time. Currently, the semantic is based on the event time. We could change the semantic to "local segment is retained for local retention time after it's uploaded to tier storage". Does that solve the problem more completely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3610670491
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2004,11 +2004,18 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long ageReferenceMs = segment.largestTimestamp();
Review Comment:
is this field name better or anchorTimestamp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on PR #22873:
URL: https://github.com/apache/kafka/pull/22873#issuecomment-5522876137
> @muralibasani thanks for this patch. BTW, would you mind adding this behavior change to `upgrade.md`?
Thank you @chia7712 for the review.
I updated PR based on the feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3730086524
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2004,11 +2004,18 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
Review Comment:
Why cache `remoteLogEnabledAndRemoteCopyEnabled`? If it is necessary, should we do it earlier?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2004,11 +2004,18 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long ageReferenceMs = segment.largestTimestamp();
Review Comment:
+1 to `anchorTimestamp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4105434127
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
I opened https://issues.apache.org/jira/browse/KAFKA-21164 for the anchor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r3731542981
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2004,11 +2004,18 @@ private int deleteRetentionMsBreachedSegments() throws IOException {
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
Review Comment:
This local var was added to give shouldDelete lambda a value to capture, rather than calling the instance method inside the lambda. but looks like it is not so effective, so removed it and have refactored it inline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "muralibasani (via GitHub)" <gi...@apache.org>.
muralibasani commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4097636082
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
Agree @junrao
I can remove this lastModified check, and rather add a warning (with check based on log.message.timestamp.after.max.ms) in LogConfig when topic is being created/altered in tiered topics flow. Also in UnifiedLog, while config is being updated, we can add similar check.
Pls let me know if this sounds ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4099270052
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
Thanks, @muralibasani !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4100354665
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
> Have we considered to use log.message.timestamp.after.max.ms to prevent future timestamps from leaking in in the first place?
Yes, this can be used to prevent future data, but the fallback is still valuable for the existing future data.
> The problem with lastModified is that it may not be accurate after reassignment or disk replacement.
Yes, the file's last modified time is not immutable. Maybe we could find another anchor, like the copy-finished timestamp in remote metadata, to replace it. I will file a JIRA for it once I'm back at my computer.
> I can remove this lastModified check
@muralibasani pardon me, do we really need to revert it? lastModified is not immutable, but it won't cause "forever". The worst case is that the deletion gets deferred, and it is a bounded delay. By contrast, future data can be stuck forever, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20609 : Use lastModified time as retention anchor for future-timestamped segments [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22873:
URL: https://github.com/apache/kafka/pull/22873#discussion_r4124814271
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/UnifiedLog.java:
##########
@@ -2000,39 +2000,34 @@ public interface DeletionCondition {
}
private int deleteRetentionMsBreachedSegments() throws IOException {
- long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled());
+ boolean remoteLogEnabledAndRemoteCopyEnabled = remoteLogEnabledAndRemoteCopyEnabled();
+ long retentionMs = UnifiedLog.localRetentionMs(config(), remoteLogEnabledAndRemoteCopyEnabled);
if (retentionMs < 0) return 0;
long startMs = time().milliseconds();
DeletionCondition shouldDelete = (segment, nextSegmentOpt) -> {
- if (startMs < segment.largestTimestamp()) {
- futureTimestampLogger.warn("{} contains future timestamp(s), making it ineligible to be deleted", segment);
+ long anchorTimestamp = segment.largestTimestamp();
+ if (startMs < anchorTimestamp) {
+ if (remoteLogEnabledAndRemoteCopyEnabled) {
+ anchorTimestamp = segment.lastModified();
Review Comment:
Got it. I have updated the ticket https://issues.apache.org/jira/browse/KAFKA-21164 to track the new semantic and the KIP :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org