You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "satishd (via GitHub)" <gi...@apache.org> on 2023/07/18 07:05:13 UTC
[GitHub] [kafka] satishd opened a new pull request, #14034: KAFKA-14483 Move/Rewrite of LocalLog to storage module.
satishd opened a new pull request, #14034:
URL: https://github.com/apache/kafka/pull/14034
KAFKA-14483 Move/Rewrite of LocalLog to storage module.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1440213705
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -2265,11 +2268,12 @@ object UnifiedLog extends Logging {
def deleteProducerSnapshots(): Unit = {
LocalLog.maybeHandleIOException(logDirFailureChannel,
parentDir,
- s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir") {
+ s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir", {
snapshotsToDelete.foreach { snapshot =>
snapshot.deleteIfExists()
}
- }
+ return;
Review Comment:
It was expecting a return for inline declaration, it is throwing a type mismatch error without that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14034:
URL: https://github.com/apache/kafka/pull/14034#issuecomment-1918983502
Resolved the recent conflicts because of trunk changes in `UnifiedLog`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1442402667
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1511,7 +1512,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
localLog.checkIfMemoryMappedBufferClosed()
// remove the segments for lookups
- localLog.removeAndDeleteSegments(segmentsToDelete, asyncDelete = true, reason)
+ localLog.removeAndDeleteSegments(segmentsToDelete.toList.asJava, true, reason)
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1511,7 +1512,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
localLog.checkIfMemoryMappedBufferClosed()
// remove the segments for lookups
- localLog.removeAndDeleteSegments(segmentsToDelete, asyncDelete = true, reason)
+ localLog.removeAndDeleteSegments(segmentsToDelete.toList.asJava, true, reason)
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1440285636
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/SegmentDeletionReason.java:
##########
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.util.List;
+
+public interface SegmentDeletionReason {
Review Comment:
`SegmentDeletionReason` has other implementations like `RetentionMsBreach`, `RetentionSizeBreach`, `StartOffsetBreach` which use `UnifiedLog`. We can take a relook at this when we refactor `UnifiedLog`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14034:
URL: https://github.com/apache/kafka/pull/14034#issuecomment-1938978627
Resolved the recent conflicts because of trunk changes in `LocalLog` by rebasing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1442403275
##########
gradle/spotbugs-exclude.xml:
##########
@@ -162,8 +162,7 @@ For a detailed description of spotbugs bug categories, see https://spotbugs.read
on the compiler to fail if the code is changed to call a method that throws Exception.
Given that, this bug pattern doesn't make sense for Scala code. -->
<Or>
- <Class name="kafka.log.Log"/>
- <Class name="kafka.log.LocalLog$"/>
+ <Class name="org.apache.kafka.storage.internals.log.LocalLog"/>
Review Comment:
Right, it is not needed, made the respective changes in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1439091103
##########
core/src/test/scala/unit/kafka/log/LocalLogTest.scala:
##########
@@ -362,8 +364,8 @@ class LocalLogTest {
}
assertEquals(5, log.segments.numberOfSegments)
assertNotEquals(10L, log.segments.activeSegment.baseOffset)
- val expected = log.segments.values.asScala.toVector
- val deleted = log.truncateFullyAndStartAt(10L)
+ val expected = new util.ArrayList(log.segments.values)
+ val deleted = StreamSupport.stream(log.truncateFullyAndStartAt(10L).spliterator(), false).collect(Collectors.toList())
Review Comment:
This can be simplified if `truncateFullyAndStartAt` returns `Collection` instead of `Iterable`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * a file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * a directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * a directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException("Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDir(dir.toPath());
Review Comment:
The Scala version called `flushDirIfExists`, why did we change it to `flushDir`?
##########
core/src/test/scala/unit/kafka/log/LocalLogTest.scala:
##########
@@ -384,10 +386,10 @@ class LocalLogTest {
assertEquals(5, log.segments.numberOfSegments)
assertEquals(12L, log.logEndOffset)
- val expected = log.segments.values(9L, log.logEndOffset + 1).asScala.toVector
+ val expected = new util.ArrayList(log.segments.values(9L, log.logEndOffset + 1))
// Truncate to an offset before the base offset of the active segment
- val deleted = log.truncateTo(7L)
- assertEquals(expected, deleted.toVector)
+ val deleted = StreamSupport.stream(log.truncateTo(7L).spliterator(), false).collect(Collectors.toList())
Review Comment:
This can be simplified if `truncateTo` returns `Collection` instead of `Iterable`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * a file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * a directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * a directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param scheduler The thread pool scheduler used for background actions
Review Comment:
Nit: should be on the next line.
##########
checkstyle/import-control-core.xml:
##########
@@ -37,6 +37,8 @@
<allow pkg="kafka.serializer" />
<allow pkg="org.apache.kafka.common" />
<allow pkg="org.mockito" class="AssignmentsManagerTest"/>
+ <allow pkg="org.apache.kafka.server.util"/>
+ <allow pkg="org.apache.kafka.storage.internals.log"/>
Review Comment:
What's the reason why we need this? A bit surprising that this would be needed for `LocalLog`, but not `LogSegment`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/SegmentDeletionReason.java:
##########
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.util.List;
+
+public interface SegmentDeletionReason {
Review Comment:
Could this be an enum if we pass `logger` via `logReason`?
##########
gradle/spotbugs-exclude.xml:
##########
@@ -162,8 +162,7 @@ For a detailed description of spotbugs bug categories, see https://spotbugs.read
on the compiler to fail if the code is changed to call a method that throws Exception.
Given that, this bug pattern doesn't make sense for Scala code. -->
<Or>
- <Class name="kafka.log.Log"/>
- <Class name="kafka.log.LocalLog$"/>
+ <Class name="org.apache.kafka.storage.internals.log.LocalLog"/>
Review Comment:
This exception was supposed to only apply to Scala code. How come we still need it?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * a file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * a directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * a directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException("Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDir(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException("Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException("Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Iterable<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException("Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * Find segments starting from the oldest until the user-supplied predicate is false.
+ * A final segment that is empty will never be returned.
+ *
+ * @param predicate A function that takes in a candidate log segment, the next higher segment
+ * (if there is one). It returns true iff the segment is deletable.
+ * @return the segments ready to be deleted
+ */
+ public Collection<LogSegment> deletableSegments(BiFunction<LogSegment, Optional<LogSegment>, Boolean> predicate) {
+ if (segments.isEmpty()) {
+ return Collections.emptyList();
+ } else {
+ ArrayList<LogSegment> deletable = new ArrayList<>();
+ Iterator<LogSegment> segmentsIterator = segments.values().iterator();
+ Optional<LogSegment> segmentOpt = nextItem(segmentsIterator);
+ while (segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ Optional<LogSegment> nextSegmentOpt = nextItem(segmentsIterator);
+ boolean isLastSegmentAndEmpty = !nextSegmentOpt.isPresent() && segment.size() == 0;
+ if (predicate.apply(segment, nextSegmentOpt) && !isLastSegmentAndEmpty) {
+ deletable.add(segment);
+ segmentOpt = nextSegmentOpt;
+ } else {
+ segmentOpt = Optional.empty();
+ }
+ }
+ return deletable;
+ }
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException("Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments =
+ segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset, long baseOffset, long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException("Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Iterable<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException("Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Iterable<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(String errorMsg,
Review Comment:
`errorMsg` should be a `Supplier<String>` so we only do string concat when needed. The Scala code did that since it used a by-name argument.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1451284145
##########
core/src/main/scala/kafka/log/LogLoader.scala:
##########
@@ -489,16 +488,16 @@ class LogLoader(
*
* @param segmentsToDelete The log segments to schedule for deletion
*/
- private def removeAndDeleteSegmentsAsync(segmentsToDelete: Iterable[LogSegment]): Unit = {
- if (segmentsToDelete.nonEmpty) {
+ private def removeAndDeleteSegmentsAsync(segmentsToDelete: java.util.Collection[LogSegment]): Unit = {
+ if (!segmentsToDelete.isEmpty) {
// Most callers hold an iterator into the `params.segments` collection and
// `removeAndDeleteSegmentAsync` mutates it by removing the deleted segment. Therefore,
// we should force materialization of the iterator here, so that results of the iteration
// remain valid and deterministic. We should also pass only the materialized view of the
// iterator to the logic that deletes the segments.
- val toDelete = segmentsToDelete.toList
- info(s"Deleting segments as part of log recovery: ${toDelete.mkString(",")}")
- toDelete.foreach { segment =>
+ val toDelete = new util.ArrayList[LogSegment](segmentsToDelete)
+ info(s"Deleting segments as part of log recovery: ${LocalLog.mkString(toDelete.iterator(), ",")}")
Review Comment:
Thanks for letting me know about `Utils.join`, I was not aware of it.
##########
core/src/test/scala/unit/kafka/log/LocalLogTest.scala:
##########
@@ -362,8 +363,8 @@ class LocalLogTest {
}
assertEquals(5, log.segments.numberOfSegments)
assertNotEquals(10L, log.segments.activeSegment.baseOffset)
- val expected = log.segments.values.asScala.toVector
- val deleted = log.truncateFullyAndStartAt(10L)
+ val expected = new util.ArrayList(log.segments.values)
+ val deleted = new util.ArrayList(log.truncateFullyAndStartAt(10L))
Review Comment:
They give different types like java `ConcurrentSkipListMap.Values` and `ArrayList`, and the equality check will fail. Wrapped around the same ArrayList instance for their equality checks.
Followed similar checks in LocalLogTest and UnifiedLogTest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14034:
URL: https://github.com/apache/kafka/pull/14034#issuecomment-1817663128
@satishd Let me know when this is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1442402484
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * a file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * a directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * a directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException("Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDir(dir.toPath());
Review Comment:
Good catch!
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * a file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * a directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * a directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException("Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDir(dir.toPath());
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1442402573
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1441,9 +1442,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
* (if there is one). It returns true iff the segment is deletable.
* @return the segments ready to be deleted
*/
- private[log] def deletableSegments(predicate: (LogSegment, Option[LogSegment]) => Boolean): Iterable[LogSegment] = {
- def isSegmentEligibleForDeletion(nextSegmentOpt: Option[LogSegment], upperBoundOffset: Long): Boolean = {
- val allowDeletionDueToLogStartOffsetIncremented = nextSegmentOpt.isDefined && logStartOffset >= nextSegmentOpt.get.baseOffset
+ private[log] def deletableSegments(predicate: (LogSegment, Optional[LogSegment]) => Boolean): Iterable[LogSegment] = {
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1441,9 +1442,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
* (if there is one). It returns true iff the segment is deletable.
* @return the segments ready to be deleted
*/
- private[log] def deletableSegments(predicate: (LogSegment, Option[LogSegment]) => Boolean): Iterable[LogSegment] = {
- def isSegmentEligibleForDeletion(nextSegmentOpt: Option[LogSegment], upperBoundOffset: Long): Boolean = {
- val allowDeletionDueToLogStartOffsetIncremented = nextSegmentOpt.isDefined && logStartOffset >= nextSegmentOpt.get.baseOffset
+ private[log] def deletableSegments(predicate: (LogSegment, Optional[LogSegment]) => Boolean): Iterable[LogSegment] = {
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1442402411
##########
checkstyle/import-control-core.xml:
##########
@@ -37,6 +37,8 @@
<allow pkg="kafka.serializer" />
<allow pkg="org.apache.kafka.common" />
<allow pkg="org.mockito" class="AssignmentsManagerTest"/>
+ <allow pkg="org.apache.kafka.server.util"/>
+ <allow pkg="org.apache.kafka.storage.internals.log"/>
Review Comment:
Right, it is not needed. I guess it was needed when that class was kept inside the core module.
##########
checkstyle/import-control-core.xml:
##########
@@ -37,6 +37,8 @@
<allow pkg="kafka.serializer" />
<allow pkg="org.apache.kafka.common" />
<allow pkg="org.mockito" class="AssignmentsManagerTest"/>
+ <allow pkg="org.apache.kafka.server.util"/>
+ <allow pkg="org.apache.kafka.storage.internals.log"/>
Review Comment:
Right, it is not needed. I guess it was needed when that class was kept inside the core module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "nizhikov (via GitHub)" <gi...@apache.org>.
nizhikov commented on PR #14034:
URL: https://github.com/apache/kafka/pull/14034#issuecomment-2147235352
Hello @satishd
Are you still working on this PR?
I can step in and finish the task if you have no time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461535963
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
Review Comment:
`existingSegments` is not a collection but `LogSegments` class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14034:
URL: https://github.com/apache/kafka/pull/14034#issuecomment-1906403295
Thanks @ijuma for the review comments. Updated the PR with the inline comments and/or with the latest commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461536622
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
Review Comment:
`createAndDeleteSegment` method it self deletes the targeted segment passed to this method. Method Javadoc with argument(segmentToDelete) clarifies that the the given segment is deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461534346
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogTruncation.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.utils.Utils;
+import org.slf4j.Logger;
+
+import java.util.List;
+
+public class LogTruncation implements SegmentDeletionReason {
+
+ private final Logger logger;
+
+ public LogTruncation(Logger logger) {
+ this.logger = logger;
+ }
+
+ @Override
+ public void logReason(List<LogSegment> toDelete) {
+ logger.info("Deleting segments as part of log truncation: {}", Utils.join(toDelete, ", "));
Review Comment:
It is done deliberately as it is better to read with that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461551624
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
+ }
+
+ // okay we are safe now, remove the swap suffix
+ for (LogSegment segment : sortedNewSegments) {
+ segment.changeFileSuffixes(SWAP_FILE_SUFFIX, "");
+ }
+ Utils.flushDir(dir.toPath());
+
+ return deletedNotReplaced;
+ }
+
+ /**
+ * Split a segment into one or more segments such that there is no offset overflow in any of them. The
+ * resulting segments will contain the exact same messages that are present in the input segment. On successful
+ * completion of this method, the input segment will be deleted and will be replaced by the resulting new segments.
+ * See replaceSegments for recovery logic, in case the broker dies in the middle of this operation.
+ * <p>
+ * Note that this method assumes we have already determined that the segment passed in contains records that cause
+ * offset overflow.
+ * <p>
+ * The split logic overloads the use of .clean files that LogCleaner typically uses to make the process of replacing
+ * the input segment with multiple new segments atomic and recoverable in the event of a crash. See replaceSegments
+ * and completeSwapOperations for the implementation to make this operation recoverable on crashes.</p>
+ *
+ * @param segment Segment to split
+ * @param existingSegments The existing segments of the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @return List of new segments that replace the input segment
+ */
+ public static SplitSegmentResult splitOverflowedSegment(LogSegment segment,
+ LogSegments existingSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ if (!isLogFile(segment.log().file())) {
+ throw new IllegalArgumentException("Cannot split file " + segment.log().file().getAbsoluteFile());
+ }
+ if (!segment.hasOverflow()) {
+ throw new IllegalArgumentException("Split operation is only permitted for segments with overflow, and the problem path is "
+ + segment.log().file().getAbsoluteFile());
+ }
+ logger.info("Splitting overflowed segment {}", segment);
+
+ List<LogSegment> newSegments = new ArrayList<>();
+ try {
+ int position = 0;
+ FileRecords sourceRecords = segment.log();
+
+ while (position < sourceRecords.sizeInBytes()) {
+ FileLogInputStream.FileChannelRecordBatch firstBatch = sourceRecords.batchesFrom(position).iterator().next();
+ LogSegment newSegment = createNewCleanedSegment(dir, config, firstBatch.baseOffset());
+ newSegments.add(newSegment);
+
+ int bytesAppended = newSegment.appendFromFile(sourceRecords, position);
+ if (bytesAppended == 0)
+ throw new IllegalStateException("Failed to append records from position " + position + " in " + segment);
+ position += bytesAppended;
+ }
+
+ // prepare new segments
+ int totalSizeOfNewSegments = 0;
+ for (LogSegment splitSegment : newSegments) {
+ splitSegment.onBecomeInactiveSegment();
+ splitSegment.flush();
+ splitSegment.setLastModified(segment.lastModified());
+ totalSizeOfNewSegments += splitSegment.log().sizeInBytes();
+ }
+ // size of all the new segments combined must equal size of the original segment
+ if (totalSizeOfNewSegments != segment.log().sizeInBytes())
+ throw new IllegalStateException("Inconsistent segment sizes after split" +
+ " before: " + segment.log().sizeInBytes() + " after: " + totalSizeOfNewSegments);
+
+ // replace old segment with new ones
+ logger.info("Replacing overflowed segment {} with split segments {}", segment, newSegments);
+ List<LogSegment> newSegmentsToAdd = Collections.unmodifiableList(newSegments);
Review Comment:
The wrapping is not done to return unmodifiable collections from `SplitSegmentResult`. But not to change the collection where it is passed to like `replaceSegments()` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1449312453
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
Review Comment:
We lost the comment for the stray dir.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
+ }
+
+ // okay we are safe now, remove the swap suffix
+ for (LogSegment segment : sortedNewSegments) {
+ segment.changeFileSuffixes(SWAP_FILE_SUFFIX, "");
+ }
+ Utils.flushDir(dir.toPath());
+
+ return deletedNotReplaced;
+ }
+
+ /**
+ * Split a segment into one or more segments such that there is no offset overflow in any of them. The
+ * resulting segments will contain the exact same messages that are present in the input segment. On successful
+ * completion of this method, the input segment will be deleted and will be replaced by the resulting new segments.
+ * See replaceSegments for recovery logic, in case the broker dies in the middle of this operation.
+ * <p>
+ * Note that this method assumes we have already determined that the segment passed in contains records that cause
+ * offset overflow.
+ * <p>
+ * The split logic overloads the use of .clean files that LogCleaner typically uses to make the process of replacing
+ * the input segment with multiple new segments atomic and recoverable in the event of a crash. See replaceSegments
+ * and completeSwapOperations for the implementation to make this operation recoverable on crashes.</p>
+ *
+ * @param segment Segment to split
+ * @param existingSegments The existing segments of the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @return List of new segments that replace the input segment
+ */
+ public static SplitSegmentResult splitOverflowedSegment(LogSegment segment,
+ LogSegments existingSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
Review Comment:
The approach of passing `logger` has different behavior than the previous approach of receiving `logPrefix`. More specifically, this allows the caller to pass a completely different logger (could be even be from a different package) while the previous approach would always use the same logger.
Is that intentional?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
Review Comment:
Existing issue, but perhaps we should add a space after `=` and before `max`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
Review Comment:
Unnecessary braces in two `if` above.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogTruncation.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.utils.Utils;
+import org.slf4j.Logger;
+
+import java.util.List;
+
+public class LogTruncation implements SegmentDeletionReason {
+
+ private final Logger logger;
+
+ public LogTruncation(Logger logger) {
+ this.logger = logger;
+ }
+
+ @Override
+ public void logReason(List<LogSegment> toDelete) {
+ logger.info("Deleting segments as part of log truncation: {}", Utils.join(toDelete, ", "));
Review Comment:
The previous implementation did not seem to include a space after `,`. Is it intentional to add it here and other similar classes?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
Review Comment:
Nit: unnecessary braces.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
Review Comment:
The previous code said `from` while this says `for`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
Review Comment:
Nit: unnecessary braces.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
Review Comment:
The previous implementation had no space after the `,`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
Review Comment:
Nit: unnecessary braces.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
+ }
+
+ // okay we are safe now, remove the swap suffix
+ for (LogSegment segment : sortedNewSegments) {
+ segment.changeFileSuffixes(SWAP_FILE_SUFFIX, "");
+ }
+ Utils.flushDir(dir.toPath());
+
+ return deletedNotReplaced;
+ }
+
+ /**
+ * Split a segment into one or more segments such that there is no offset overflow in any of them. The
+ * resulting segments will contain the exact same messages that are present in the input segment. On successful
+ * completion of this method, the input segment will be deleted and will be replaced by the resulting new segments.
+ * See replaceSegments for recovery logic, in case the broker dies in the middle of this operation.
+ * <p>
+ * Note that this method assumes we have already determined that the segment passed in contains records that cause
+ * offset overflow.
+ * <p>
+ * The split logic overloads the use of .clean files that LogCleaner typically uses to make the process of replacing
+ * the input segment with multiple new segments atomic and recoverable in the event of a crash. See replaceSegments
+ * and completeSwapOperations for the implementation to make this operation recoverable on crashes.</p>
+ *
+ * @param segment Segment to split
+ * @param existingSegments The existing segments of the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @return List of new segments that replace the input segment
+ */
+ public static SplitSegmentResult splitOverflowedSegment(LogSegment segment,
+ LogSegments existingSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ if (!isLogFile(segment.log().file())) {
+ throw new IllegalArgumentException("Cannot split file " + segment.log().file().getAbsoluteFile());
+ }
+ if (!segment.hasOverflow()) {
+ throw new IllegalArgumentException("Split operation is only permitted for segments with overflow, and the problem path is "
+ + segment.log().file().getAbsoluteFile());
+ }
Review Comment:
Nit: braces are not needed for either of the checks above.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
+ }
+
+ // okay we are safe now, remove the swap suffix
+ for (LogSegment segment : sortedNewSegments) {
+ segment.changeFileSuffixes(SWAP_FILE_SUFFIX, "");
+ }
+ Utils.flushDir(dir.toPath());
+
+ return deletedNotReplaced;
+ }
+
+ /**
+ * Split a segment into one or more segments such that there is no offset overflow in any of them. The
+ * resulting segments will contain the exact same messages that are present in the input segment. On successful
+ * completion of this method, the input segment will be deleted and will be replaced by the resulting new segments.
+ * See replaceSegments for recovery logic, in case the broker dies in the middle of this operation.
+ * <p>
+ * Note that this method assumes we have already determined that the segment passed in contains records that cause
+ * offset overflow.
+ * <p>
+ * The split logic overloads the use of .clean files that LogCleaner typically uses to make the process of replacing
+ * the input segment with multiple new segments atomic and recoverable in the event of a crash. See replaceSegments
+ * and completeSwapOperations for the implementation to make this operation recoverable on crashes.</p>
+ *
+ * @param segment Segment to split
+ * @param existingSegments The existing segments of the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @return List of new segments that replace the input segment
+ */
+ public static SplitSegmentResult splitOverflowedSegment(LogSegment segment,
+ LogSegments existingSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ if (!isLogFile(segment.log().file())) {
+ throw new IllegalArgumentException("Cannot split file " + segment.log().file().getAbsoluteFile());
+ }
+ if (!segment.hasOverflow()) {
+ throw new IllegalArgumentException("Split operation is only permitted for segments with overflow, and the problem path is "
+ + segment.log().file().getAbsoluteFile());
+ }
+ logger.info("Splitting overflowed segment {}", segment);
+
+ List<LogSegment> newSegments = new ArrayList<>();
+ try {
+ int position = 0;
+ FileRecords sourceRecords = segment.log();
+
+ while (position < sourceRecords.sizeInBytes()) {
+ FileLogInputStream.FileChannelRecordBatch firstBatch = sourceRecords.batchesFrom(position).iterator().next();
+ LogSegment newSegment = createNewCleanedSegment(dir, config, firstBatch.baseOffset());
+ newSegments.add(newSegment);
+
+ int bytesAppended = newSegment.appendFromFile(sourceRecords, position);
+ if (bytesAppended == 0)
+ throw new IllegalStateException("Failed to append records from position " + position + " in " + segment);
+ position += bytesAppended;
+ }
+
+ // prepare new segments
+ int totalSizeOfNewSegments = 0;
+ for (LogSegment splitSegment : newSegments) {
+ splitSegment.onBecomeInactiveSegment();
+ splitSegment.flush();
+ splitSegment.setLastModified(segment.lastModified());
+ totalSizeOfNewSegments += splitSegment.log().sizeInBytes();
+ }
+ // size of all the new segments combined must equal size of the original segment
+ if (totalSizeOfNewSegments != segment.log().sizeInBytes())
+ throw new IllegalStateException("Inconsistent segment sizes after split" +
+ " before: " + segment.log().sizeInBytes() + " after: " + totalSizeOfNewSegments);
+
+ // replace old segment with new ones
+ logger.info("Replacing overflowed segment {} with split segments {}", segment, newSegments);
+ List<LogSegment> newSegmentsToAdd = Collections.unmodifiableList(newSegments);
Review Comment:
It's more natural for `SplitSegmentResult` to do the wrapping in its constructor.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
+ }
+
+ // okay we are safe now, remove the swap suffix
+ for (LogSegment segment : sortedNewSegments) {
+ segment.changeFileSuffixes(SWAP_FILE_SUFFIX, "");
+ }
Review Comment:
Nit: unnecessary braces.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
Review Comment:
Nit: perhaps add javadoc indicating that this is equivalent to `roll(OptionalLong.empty())` and a note to see the overloaded variant for more details.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
Review Comment:
Nit: unnecessary braces here.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
Review Comment:
Can we use `addAll` instead?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
Review Comment:
Instead of generating the intermediate collection, it seems more efficient to add each item directly to `abortedTransaction`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
Review Comment:
Again, we should perhaps add a space after `=` (existing issue).
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
Review Comment:
There is a `+` inside the string and perhaps we should add a space after `=` (existing issue).
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
Review Comment:
This comment (which also exists in the old code) is a bit odd. Where is the code for "then delete the old last segment..."?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
Review Comment:
`_maxEntries` should be `maxEntries` now? Worth checking if there are any other comments that reference variable names with `_`.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
Review Comment:
Why are we using a different overload of `open` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1448871794
##########
core/src/main/scala/kafka/log/LogLoader.scala:
##########
@@ -489,16 +488,16 @@ class LogLoader(
*
* @param segmentsToDelete The log segments to schedule for deletion
*/
- private def removeAndDeleteSegmentsAsync(segmentsToDelete: Iterable[LogSegment]): Unit = {
- if (segmentsToDelete.nonEmpty) {
+ private def removeAndDeleteSegmentsAsync(segmentsToDelete: java.util.Collection[LogSegment]): Unit = {
+ if (!segmentsToDelete.isEmpty) {
// Most callers hold an iterator into the `params.segments` collection and
// `removeAndDeleteSegmentAsync` mutates it by removing the deleted segment. Therefore,
// we should force materialization of the iterator here, so that results of the iteration
// remain valid and deterministic. We should also pass only the materialized view of the
// iterator to the logic that deletes the segments.
- val toDelete = segmentsToDelete.toList
- info(s"Deleting segments as part of log recovery: ${toDelete.mkString(",")}")
- toDelete.foreach { segment =>
+ val toDelete = new util.ArrayList[LogSegment](segmentsToDelete)
+ info(s"Deleting segments as part of log recovery: ${LocalLog.mkString(toDelete.iterator(), ",")}")
Review Comment:
Can you not use `Utils.join` instead of creating a new implementation here?
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -2246,30 +2251,31 @@ object UnifiedLog extends Logging {
config: LogConfig,
scheduler: Scheduler,
logDirFailureChannel: LogDirFailureChannel,
- logPrefix: String): SplitSegmentResult = {
- LocalLog.splitOverflowedSegment(segment, existingSegments, dir, topicPartition, config, scheduler, logDirFailureChannel, logPrefix)
+ logger: Logger): SplitSegmentResult = {
+ LocalLog.splitOverflowedSegment(segment, existingSegments, dir, topicPartition, config, scheduler, logDirFailureChannel, logger)
}
- private[log] def deleteProducerSnapshots(segments: Iterable[LogSegment],
+ private[log] def deleteProducerSnapshots(segments: util.Collection[LogSegment],
producerStateManager: ProducerStateManager,
asyncDelete: Boolean,
scheduler: Scheduler,
config: LogConfig,
logDirFailureChannel: LogDirFailureChannel,
parentDir: String,
topicPartition: TopicPartition): Unit = {
- val snapshotsToDelete = segments.flatMap { segment =>
- producerStateManager.removeAndMarkSnapshotForDeletion(segment.baseOffset).asScala
- }
+ val snapshotsToDelete = segments.stream().flatMap { segment =>
+ val snapshotOptional = producerStateManager.removeAndMarkSnapshotForDeletion(segment.baseOffset)
+ if (snapshotOptional.isPresent) util.stream.Stream.of[SnapshotFile](snapshotOptional.get) else util.stream.Stream.empty[SnapshotFile]
+ }.collect(Collectors.toList[SnapshotFile])
def deleteProducerSnapshots(): Unit = {
- LocalLog.maybeHandleIOException(logDirFailureChannel,
- parentDir,
- s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir") {
- snapshotsToDelete.foreach { snapshot =>
- snapshot.deleteIfExists()
- }
- }
+ LocalLog.maybeHandleIOException(logDirFailureChannel, parentDir,
+ () => s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir",
+ () => {
Review Comment:
I don't think you need the block since there is only one expression inside it.
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -3971,10 +3973,10 @@ class UnifiedLogTest {
))
log.appendAsLeader(records, leaderEpoch = 0)
log.maybeIncrementHighWatermark(log.logEndOffsetMetadata)
- val deletable = log.deletableSegments((_: LogSegment, _: Option[LogSegment]) => true)
- val expected = log.logSegments.asScala.toList
- assertEquals(10, expected.length)
- assertEquals(expected, deletable.toList)
+ val deletable = new util.ArrayList(log.deletableSegments((_: LogSegment, _: Optional[LogSegment]) => true))
+ val expected = new util.ArrayList(log.logSegments)
Review Comment:
Are these copies needed?
##########
core/src/test/scala/unit/kafka/log/LocalLogTest.scala:
##########
@@ -362,8 +363,8 @@ class LocalLogTest {
}
assertEquals(5, log.segments.numberOfSegments)
assertNotEquals(10L, log.segments.activeSegment.baseOffset)
- val expected = log.segments.values.asScala.toVector
- val deleted = log.truncateFullyAndStartAt(10L)
+ val expected = new util.ArrayList(log.segments.values)
+ val deleted = new util.ArrayList(log.truncateFullyAndStartAt(10L))
Review Comment:
Why is this needed?
##########
core/src/main/scala/kafka/log/LogCleaner.scala:
##########
@@ -1245,8 +1246,8 @@ private[log] class CleanedTransactionMetadata {
*
* @param abortedTransactions The new found aborted transactions to add
*/
- def addAbortedTransactions(abortedTransactions: List[AbortedTxn]): Unit = {
- this.abortedTransactions ++= abortedTransactions
+ def addAbortedTransactions(abortedTransactions: util.List[AbortedTxn]): Unit = {
+ abortedTransactions.forEach(abortedTxn => this.abortedTransactions += abortedTxn)
Review Comment:
Can this be `this.abortedTransactions ++= abortedTransactions.asScala`? Adding in bulk is usually more efficient.
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -4015,7 +4017,7 @@ class UnifiedLogTest {
true
})
assertEquals(10L, log.logSegments.size())
- assertEquals(log.nonActiveLogSegmentsFrom(0L).asScala.toSeq, deletableSegments.toSeq)
+ assertEquals(new util.ArrayList(log.nonActiveLogSegmentsFrom(0L)), deletableSegments)
Review Comment:
Is this copy needed?
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -3951,18 +3953,18 @@ class UnifiedLogTest {
assertEquals(10, log.logSegments.size())
{
- val deletable = log.deletableSegments(
- (segment: LogSegment, _: Option[LogSegment]) => segment.baseOffset <= 5)
- val expected = log.nonActiveLogSegmentsFrom(0L).asScala.filter(segment => segment.baseOffset <= 5).toList
- assertEquals(6, expected.length)
- assertEquals(expected, deletable.toList)
+ val deletable = new util.ArrayList(
+ log.deletableSegments((segment: LogSegment, _: Optional[LogSegment]) => segment.baseOffset <= 5))
+ val expected = log.nonActiveLogSegmentsFrom(0L).stream().filter(segment => segment.baseOffset <= 5).collect(Collectors.toList())
+ assertEquals(6, expected.size)
+ assertEquals(expected, deletable)
}
{
- val deletable = log.deletableSegments((_: LogSegment, _: Option[LogSegment]) => true)
- val expected = log.nonActiveLogSegmentsFrom(0L).asScala.toList
- assertEquals(9, expected.length)
- assertEquals(expected, deletable.toList)
+ val deletable = new util.ArrayList(log.deletableSegments((_: LogSegment, _: Optional[LogSegment]) => true))
+ val expected = new util.ArrayList(log.nonActiveLogSegmentsFrom(0L))
Review Comment:
Are these copies needed?
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -2246,30 +2251,31 @@ object UnifiedLog extends Logging {
config: LogConfig,
scheduler: Scheduler,
logDirFailureChannel: LogDirFailureChannel,
- logPrefix: String): SplitSegmentResult = {
- LocalLog.splitOverflowedSegment(segment, existingSegments, dir, topicPartition, config, scheduler, logDirFailureChannel, logPrefix)
+ logger: Logger): SplitSegmentResult = {
+ LocalLog.splitOverflowedSegment(segment, existingSegments, dir, topicPartition, config, scheduler, logDirFailureChannel, logger)
}
- private[log] def deleteProducerSnapshots(segments: Iterable[LogSegment],
+ private[log] def deleteProducerSnapshots(segments: util.Collection[LogSegment],
producerStateManager: ProducerStateManager,
asyncDelete: Boolean,
scheduler: Scheduler,
config: LogConfig,
logDirFailureChannel: LogDirFailureChannel,
parentDir: String,
topicPartition: TopicPartition): Unit = {
- val snapshotsToDelete = segments.flatMap { segment =>
- producerStateManager.removeAndMarkSnapshotForDeletion(segment.baseOffset).asScala
- }
+ val snapshotsToDelete = segments.stream().flatMap { segment =>
+ val snapshotOptional = producerStateManager.removeAndMarkSnapshotForDeletion(segment.baseOffset)
+ if (snapshotOptional.isPresent) util.stream.Stream.of[SnapshotFile](snapshotOptional.get) else util.stream.Stream.empty[SnapshotFile]
Review Comment:
Nit: does `snapshotOptional.map(Stream::of).orElseGet(Stream::empty);` work? Also, it can probably be inlined (i.e. no need for the variable `snapshotOptional`.
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -3951,18 +3953,18 @@ class UnifiedLogTest {
assertEquals(10, log.logSegments.size())
{
- val deletable = log.deletableSegments(
- (segment: LogSegment, _: Option[LogSegment]) => segment.baseOffset <= 5)
- val expected = log.nonActiveLogSegmentsFrom(0L).asScala.filter(segment => segment.baseOffset <= 5).toList
- assertEquals(6, expected.length)
- assertEquals(expected, deletable.toList)
+ val deletable = new util.ArrayList(
Review Comment:
Do we actually need this copy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14034:
URL: https://github.com/apache/kafka/pull/14034#issuecomment-1759964557
@satishd Since we now have #14529, do you think you'll be able to pick this up again soon?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1439152043
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1441,9 +1442,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
* (if there is one). It returns true iff the segment is deletable.
* @return the segments ready to be deleted
*/
- private[log] def deletableSegments(predicate: (LogSegment, Option[LogSegment]) => Boolean): Iterable[LogSegment] = {
- def isSegmentEligibleForDeletion(nextSegmentOpt: Option[LogSegment], upperBoundOffset: Long): Boolean = {
- val allowDeletionDueToLogStartOffsetIncremented = nextSegmentOpt.isDefined && logStartOffset >= nextSegmentOpt.get.baseOffset
+ private[log] def deletableSegments(predicate: (LogSegment, Optional[LogSegment]) => Boolean): Iterable[LogSegment] = {
Review Comment:
We should change this to return `java.util.Collection[LogSegment]` to avoid unnecessary conversions.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1511,7 +1512,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
localLog.checkIfMemoryMappedBufferClosed()
// remove the segments for lookups
- localLog.removeAndDeleteSegments(segmentsToDelete, asyncDelete = true, reason)
+ localLog.removeAndDeleteSegments(segmentsToDelete.toList.asJava, true, reason)
Review Comment:
This conversion can be avoided if we make the change to the method signature suggested above.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1564,8 +1565,10 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private def deleteLogStartOffsetBreachedSegments(): Int = {
- def shouldDelete(segment: LogSegment, nextSegmentOpt: Option[LogSegment]): Boolean = {
- nextSegmentOpt.exists(_.baseOffset <= (if (remoteLogEnabled()) localLogStartOffset() else logStartOffset))
+ def shouldDelete(segment: LogSegment, nextSegmentOpt: Optional[LogSegment]): Boolean = {
+ if (nextSegmentOpt.isPresent)
+ nextSegmentOpt.get().baseOffset <= (if (remoteLogEnabled()) localLogStartOffset() else logStartOffset)
+ else false
Review Comment:
Nit: you can do `nextSegmentOpt.isPresent && nextSegmentOpt.get...`
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1245,7 +1246,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private[log] def collectAbortedTransactions(startOffset: Long, upperBoundOffset: Long): List[AbortedTxn] = {
Review Comment:
We should change this to return `java.util.Collection` or `java.util.List` to avoid unnecessary conversions.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -2265,11 +2268,12 @@ object UnifiedLog extends Logging {
def deleteProducerSnapshots(): Unit = {
LocalLog.maybeHandleIOException(logDirFailureChannel,
parentDir,
- s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir") {
+ s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir", {
snapshotsToDelete.foreach { snapshot =>
snapshot.deleteIfExists()
}
- }
+ return;
Review Comment:
Hmm, it is a bit odd that a `return` with no value is required for scala code. Is this right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1442402745
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1245,7 +1246,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private[log] def collectAbortedTransactions(startOffset: Long, upperBoundOffset: Long): List[AbortedTxn] = {
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1245,7 +1246,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private[log] def collectAbortedTransactions(startOffset: Long, upperBoundOffset: Long): List[AbortedTxn] = {
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -2265,11 +2268,12 @@ object UnifiedLog extends Logging {
def deleteProducerSnapshots(): Unit = {
LocalLog.maybeHandleIOException(logDirFailureChannel,
parentDir,
- s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir") {
+ s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir", {
snapshotsToDelete.foreach { snapshot =>
snapshot.deleteIfExists()
}
- }
+ return;
Review Comment:
It was expecting a return for inline declaration, it is throwing a type mismatch error without that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461532356
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ // We need to update the segment base offset and append position data of the metadata when log rolls.
+ // The next offset should not change.
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+
+ return newSegment;
+ });
+ }
+
+ /**
+ * Delete all data in the local log and start at the new offset.
+ *
+ * @param newOffset The new offset to start the log with
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateFullyAndStartAt(long newOffset) {
+ return maybeHandleIOException(
+ () -> "Error while truncating the entire log for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.debug("Truncate and start at offset {}", newOffset);
+ checkIfMemoryMappedBufferClosed();
+ List<LogSegment> segmentsToDelete = new ArrayList<>(segments.values());
+
+ if (!segmentsToDelete.isEmpty()) {
+ removeAndDeleteSegments(segmentsToDelete.subList(0, segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ // Use createAndDeleteSegment() to create new segment first and then delete the old last segment to prevent missing
+ // active segment during the deletion process
+ createAndDeleteSegment(newOffset, segmentsToDelete.get(segmentsToDelete.size() - 1), true, new LogTruncation(this.logger));
+ }
+
+ updateLogEndOffset(newOffset);
+
+ return segmentsToDelete;
+ });
+ }
+
+ /**
+ * Truncate this log so that it ends with the greatest offset < targetOffset.
+ *
+ * @param targetOffset The offset to truncate to, an upper bound on all offsets in the log after truncation is complete.
+ * @return the list of segments that were scheduled for deletion
+ */
+ public Collection<LogSegment> truncateTo(long targetOffset) throws IOException {
+ Collection<LogSegment> deletableSegments = segments.filter(segment -> segment.baseOffset() > targetOffset);
+ removeAndDeleteSegments(deletableSegments, true, new LogTruncation(this.logger));
+ segments.activeSegment().truncateTo(targetOffset);
+ updateLogEndOffset(targetOffset);
+ return deletableSegments;
+ }
+
+ private <T> T maybeHandleIOException(Supplier<String> errorMsgSupplier,
+ StorageAction<T, IOException> func) {
+ return maybeHandleIOException(logDirFailureChannel, parentDir, errorMsgSupplier, func);
+ }
+
+ /**
+ * Wraps the value of iterator.next() in an Optional instance.
+ *
+ * @param iterator given iterator to iterate over
+ * @return if a next element exists, Optional#empty otherwise.
+ * @param <T> the type of object held within the iterator
+ */
+ public static <T> Optional<T> nextItem(Iterator<T> iterator) {
+ return iterator.hasNext() ? Optional.of(iterator.next()) : Optional.empty();
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for async deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-delete".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logDeleteDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, DELETE_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a new directory name in the following format: "${topic}-${partitionId}.${uniqueId}${suffix}".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ static String logDirNameWithSuffixCappedLength(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ String fullSuffix = "-" + topicPartition.partition() + "." + uniqueId + suffix;
+ int prefixLength = Math.min(topicPartition.topic().length(), 255 - fullSuffix.length());
+ return topicPartition.topic().substring(0, prefixLength) + fullSuffix;
+ }
+
+ /**
+ * Return a directory name to rename the log directory to for stray partition deletion.
+ * The name will be in the following format: "topic-partitionId.uniqueId-stray".
+ * If the topic name is too long, it will be truncated to prevent the total name
+ * from exceeding 255 characters.
+ */
+ public static String logStrayDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffixCappedLength(topicPartition, STRAY_DIR_SUFFIX);
+ }
+
+ /**
+ * Return a future directory name for the given topic partition. The name will be in the following
+ * format: topic-partition.uniqueId-future where topic, partition and uniqueId are variables.
+ */
+ public static String logFutureDirName(TopicPartition topicPartition) {
+ return logDirNameWithSuffix(topicPartition, FUTURE_DIR_SUFFIX);
+ }
+
+ static String logDirNameWithSuffix(TopicPartition topicPartition, String suffix) {
+ String uniqueId = java.util.UUID.randomUUID().toString().replaceAll("-", "");
+ return logDirName(topicPartition) + "." + uniqueId + suffix;
+ }
+
+ /**
+ * Return a directory name for the given topic partition. The name will be in the following
+ * format: topic-partition where topic, partition are variables.
+ */
+ public static String logDirName(TopicPartition topicPartition) {
+ return topicPartition.topic() + "-" + topicPartition.partition();
+ }
+
+ /**
+ * Parse the topic and partition out of the directory name of a log
+ */
+ public static TopicPartition parseTopicPartitionName(File dir) {
+ if (dir == null) throw new KafkaException("dir should not be null");
+
+ Function<File, KafkaException> exceptionFn
+ = file -> new KafkaException("Found directory " + dir.getAbsolutePath() + ", '" + dir.getName() +
+ "' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).\n"
+ + "Kafka's log directories (and children) should only contain Kafka topic data.");
+
+ String dirName = dir.getName();
+ if (dirName == null || dirName.isEmpty() || !dirName.contains("-"))
+ throw exceptionFn.apply(dir);
+
+ if (dirName.endsWith(DELETE_DIR_SUFFIX) && !DELETE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(FUTURE_DIR_SUFFIX) && !FUTURE_DIR_PATTERN.matcher(dirName).matches() ||
+ dirName.endsWith(STRAY_DIR_SUFFIX) && !STRAY_DIR_PATTERN.matcher(dirName).matches())
+ throw exceptionFn.apply(dir);
+
+ String name = dirName.endsWith(DELETE_DIR_SUFFIX) || dirName.endsWith(FUTURE_DIR_SUFFIX) || dirName.endsWith(STRAY_DIR_SUFFIX)
+ ? dirName.substring(0, dirName.lastIndexOf('.')) : dirName;
+
+ int index = name.lastIndexOf('-');
+ String topic = name.substring(0, index);
+ String partitionString = name.substring(index + 1);
+
+ if (topic.isEmpty() || partitionString.isEmpty()) throw exceptionFn.apply(dir);
+
+ try {
+ int partition = Integer.parseInt(partitionString);
+ return new TopicPartition(topic, partition);
+ } catch (NumberFormatException ex) {
+ throw exceptionFn.apply(dir);
+ }
+ }
+
+ public static boolean isIndexFile(File file) {
+ String fileName = file.getName();
+ return fileName.endsWith(LogFileUtils.INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TIME_INDEX_FILE_SUFFIX)
+ || fileName.endsWith(LogFileUtils.TXN_INDEX_FILE_SUFFIX);
+ }
+
+ public static boolean isLogFile(File file) {
+ return file.getPath().endsWith(LogFileUtils.LOG_FILE_SUFFIX);
+ }
+
+ FetchDataInfo emptyFetchDataInfo(LogOffsetMetadata fetchOffsetMetadata, boolean includeAbortedTxns) {
+ Optional<List<FetchResponseData.AbortedTransaction>> abortedTransactions
+ = includeAbortedTxns ? Optional.of(Collections.emptyList()) : Optional.empty();
+
+ return new FetchDataInfo(fetchOffsetMetadata, MemoryRecords.EMPTY, false, abortedTransactions);
+ }
+
+ public static LogSegment createNewCleanedSegment(File dir, LogConfig logConfig, long baseOffset) throws IOException {
+ LogSegment.deleteIfExists(dir, baseOffset, CLEANED_FILE_SUFFIX);
+ return LogSegment.open(dir,
+ baseOffset,
+ logConfig,
+ Time.SYSTEM,
+ false,
+ logConfig.initFileSize(),
+ logConfig.preallocate,
+ CLEANED_FILE_SUFFIX);
+ }
+
+ /**
+ * Perform physical deletion of the index and log files for the given segment.
+ * Prior to the deletion, the index and log files are renamed by appending .deleted to the
+ * respective file name. Allows these files to be optionally deleted asynchronously.
+ * <p>
+ * This method assumes that the file exists. It does not need to convert IOException
+ * (thrown from changeFileSuffixes) to KafkaStorageException because it is either called before
+ * all logs are loaded or the caller will catch and handle IOException.
+ *
+ * @param segmentsToDelete The segments to be deleted
+ * @param asyncDelete If true, the deletion of the segments is done asynchronously
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @throws IOException if the file can't be renamed and still exists
+ */
+ public static void deleteSegmentFiles(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
+ for (LogSegment segment : segmentsToDelete) {
+ if (!segment.hasSuffix(LogFileUtils.DELETED_FILE_SUFFIX))
+ segment.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ Runnable deleteSegments = () -> {
+ logger.info("Deleting segment files {}", Utils.join(segmentsToDelete, ", "));
+ String parentDir = dir.getParent();
+ maybeHandleIOException(logDirFailureChannel, parentDir,
+ () -> "Error while deleting segments for " + topicPartition + " in dir " + parentDir,
+ () -> {
+ for (LogSegment segment : segmentsToDelete) {
+ segment.deleteIfExists();
+ }
+ return null;
+ });
+ };
+
+ if (asyncDelete) {
+ scheduler.scheduleOnce("delete-file", deleteSegments, config.fileDeleteDelayMs);
+ } else {
+ deleteSegments.run();
+ }
+ }
+
+ /**
+ * Swap one or more new segment in place and delete one or more existing segments in a crash-safe
+ * manner. The old segments will be asynchronously deleted.
+ * <p>
+ * This method does not need to convert IOException to KafkaStorageException because it is either
+ * called before all logs are loaded or the caller will catch and handle IOException
+ * <p>
+ * The sequence of operations is:
+ * <p>
+ * - Cleaner creates one or more new segments with suffix .cleaned and invokes replaceSegments() on
+ * the Log instance. If broker crashes at this point, the clean-and-swap operation is aborted and
+ * the .cleaned files are deleted on recovery in LogLoader.
+ * - New segments are renamed .swap. If the broker crashes before all segments were renamed to .swap, the
+ * clean-and-swap operation is aborted - .cleaned as well as .swap files are deleted on recovery in
+ * in LogLoader. We detect this situation by maintaining a specific order in which files are renamed
+ * from .cleaned to .swap. Basically, files are renamed in descending order of offsets. On recovery,
+ * all .swap files whose offset is greater than the minimum-offset .clean file are deleted.
+ * - If the broker crashes after all new segments were renamed to .swap, the operation is completed,
+ * the swap operation is resumed on recovery as described in the next step.
+ * - Old segment files are renamed to .deleted and asynchronous delete is scheduled. If the broker
+ * crashes, any .deleted files left behind are deleted on recovery in LogLoader.
+ * replaceSegments() is then invoked to complete the swap with newSegment recreated from the
+ * .swap file and oldSegments containing segments which were not renamed before the crash.
+ * - Swap segment(s) are renamed to replace the existing segments, completing this operation.
+ * If the broker crashes, any .deleted files which may be left behind are deleted
+ * on recovery in LogLoader.
+ *
+ * @param existingSegments The existing segments of the log
+ * @param newSegments The new log segment to add to the log
+ * @param oldSegments The old log segments to delete from the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @param isRecoveredSwapFile true if the new segment was created from a swap file during recovery after a crash
+ */
+ public static List<LogSegment> replaceSegments(LogSegments existingSegments,
+ List<LogSegment> newSegments,
+ List<LogSegment> oldSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger,
+ boolean isRecoveredSwapFile) throws IOException {
+ List<LogSegment> sortedNewSegments = new ArrayList<>(newSegments);
+ sortedNewSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ // Some old segments may have been removed from index and scheduled for async deletion after the caller reads segments
+ // but before this method is executed. We want to filter out those segments to avoid calling deleteSegmentFiles()
+ // multiple times for the same segment.
+ List<LogSegment> sortedOldSegments = oldSegments.stream()
+ .filter(seg -> existingSegments.contains(seg.baseOffset()))
+ .sorted(Comparator.comparingLong(LogSegment::baseOffset))
+ .collect(Collectors.toList());
+
+ // need to do this in two phases to be crash safe AND do the delete asynchronously
+ // if we crash in the middle of this we complete the swap in loadSegments()
+ List<LogSegment> reversedSegmentsList = new ArrayList<>(sortedNewSegments);
+ Collections.reverse(reversedSegmentsList);
+ if (!isRecoveredSwapFile) {
+ for (LogSegment logSegment : reversedSegmentsList) {
+ logSegment.changeFileSuffixes(CLEANED_FILE_SUFFIX, SWAP_FILE_SUFFIX);
+ }
+ }
+ for (LogSegment logSegment : reversedSegmentsList) {
+ existingSegments.add(logSegment);
+ }
+
+ Set<Long> newSegmentBaseOffsets = sortedNewSegments.stream().map(LogSegment::baseOffset)
+ .collect(Collectors.toSet());
+
+ // delete the old files
+ List<LogSegment> deletedNotReplaced = new ArrayList<>();
+ for (LogSegment seg : sortedOldSegments) {
+ // remove the index entry
+ if (seg.baseOffset() != sortedNewSegments.get(0).baseOffset())
+ existingSegments.remove(seg.baseOffset());
+ deleteSegmentFiles(
+ Collections.singletonList(seg),
+ true,
+ dir,
+ topicPartition,
+ config,
+ scheduler,
+ logDirFailureChannel,
+ logger);
+ if (!newSegmentBaseOffsets.contains(seg.baseOffset())) {
+ deletedNotReplaced.add(seg);
+ }
+ }
+
+ // okay we are safe now, remove the swap suffix
+ for (LogSegment segment : sortedNewSegments) {
+ segment.changeFileSuffixes(SWAP_FILE_SUFFIX, "");
+ }
+ Utils.flushDir(dir.toPath());
+
+ return deletedNotReplaced;
+ }
+
+ /**
+ * Split a segment into one or more segments such that there is no offset overflow in any of them. The
+ * resulting segments will contain the exact same messages that are present in the input segment. On successful
+ * completion of this method, the input segment will be deleted and will be replaced by the resulting new segments.
+ * See replaceSegments for recovery logic, in case the broker dies in the middle of this operation.
+ * <p>
+ * Note that this method assumes we have already determined that the segment passed in contains records that cause
+ * offset overflow.
+ * <p>
+ * The split logic overloads the use of .clean files that LogCleaner typically uses to make the process of replacing
+ * the input segment with multiple new segments atomic and recoverable in the event of a crash. See replaceSegments
+ * and completeSwapOperations for the implementation to make this operation recoverable on crashes.</p>
+ *
+ * @param segment Segment to split
+ * @param existingSegments The existing segments of the log
+ * @param dir The directory in which the log will reside
+ * @param topicPartition The topic
+ * @param config The log configuration settings
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param logDirFailureChannel The LogDirFailureChannel to asynchronously handle log dir failure
+ * @param logger The logger to log messages
+ * @return List of new segments that replace the input segment
+ */
+ public static SplitSegmentResult splitOverflowedSegment(LogSegment segment,
+ LogSegments existingSegments,
+ File dir,
+ TopicPartition topicPartition,
+ LogConfig config,
+ Scheduler scheduler,
+ LogDirFailureChannel logDirFailureChannel,
+ Logger logger) throws IOException {
Review Comment:
Yes, it was intentional. Earlier, `logPrefix` was passed so that the respective method callers could check the respective log statements in `LocalLog` logger logs by filtering with the passed log-prefixes.
So, I thought it is better to print these logs with the respective loggers instead of sending their prefixes.
If the caller method uses a different logger then it would have been hard for users to shuffle between their logs and `LocalLog` class logs. With the new change, they will get all the related logs in their respective logger logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1451284647
##########
core/src/test/scala/unit/kafka/log/UnifiedLogTest.scala:
##########
@@ -3951,18 +3953,18 @@ class UnifiedLogTest {
assertEquals(10, log.logSegments.size())
{
- val deletable = log.deletableSegments(
- (segment: LogSegment, _: Option[LogSegment]) => segment.baseOffset <= 5)
- val expected = log.nonActiveLogSegmentsFrom(0L).asScala.filter(segment => segment.baseOffset <= 5).toList
- assertEquals(6, expected.length)
- assertEquals(expected, deletable.toList)
+ val deletable = new util.ArrayList(
Review Comment:
This is strictly not needed as it returns `ArrayList` in this test's successful case scenario but the same method can return `EmptyList` instance also in other case. I wanted the test to fail with details about the elements instead of type mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1451284748
##########
core/src/main/scala/kafka/log/LogCleaner.scala:
##########
@@ -1245,8 +1246,8 @@ private[log] class CleanedTransactionMetadata {
*
* @param abortedTransactions The new found aborted transactions to add
*/
- def addAbortedTransactions(abortedTransactions: List[AbortedTxn]): Unit = {
- this.abortedTransactions ++= abortedTransactions
+ def addAbortedTransactions(abortedTransactions: util.List[AbortedTxn]): Unit = {
+ abortedTransactions.forEach(abortedTxn => this.abortedTransactions += abortedTxn)
Review Comment:
Wanted to avoid scala conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461537909
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
+ logger.warn("Trying to roll a new log segment with start offset {} =max(provided offset = {}, LEO = {}) while it already exists and is active with size 0. Size of time index: {}, size of offset index: {}.",
+ newOffset, expectedNextOffset, logEndOffset(), activeSegment.timeIndex().entries(), activeSegment.offsetIndex().entries());
+ LogSegment newSegment = createAndDeleteSegment(newOffset, activeSegment, true, new LogRoll(this.logger));
+ updateLogEndOffset(nextOffsetMetadata.messageOffset);
+ logger.info("Rolled new log segment at offset {} in {} ms.", newOffset, time.hiResClockMs() - start);
+ return newSegment;
+ } else {
+ throw new KafkaException("Trying to roll a new log segment for topic partition "
+ + topicPartition + " with start offset " + newOffset + " + =max(provided offset = "
+ + expectedNextOffset + ", LEO = " + logEndOffset() + " while it already exists. Existing segment is "
+ + segments.get(newOffset) + ".");
+ }
+ } else if (!segments.isEmpty() && newOffset < activeSegment.baseOffset()) {
+ throw new KafkaException("Trying to roll a new log segment for topic partition " + topicPartition
+ + " with start offset " + newOffset + " =max(provided offset = " + expectedNextOffset
+ + ", LEO = " + logEndOffset() + ") lower than start offset of the active segment " + activeSegment);
+ } else {
+ File offsetIdxFile = LogFileUtils.offsetIndexFile(dir, newOffset);
+ File timeIdxFile = LogFileUtils.timeIndexFile(dir, newOffset);
+ File txnIdxFile = LogFileUtils.transactionIndexFile(dir, newOffset);
+
+ for (File file : Arrays.asList(logFile, offsetIdxFile, timeIdxFile, txnIdxFile)) {
+ if (file.exists()) {
+ logger.warn("Newly rolled segment file {} already exists; deleting it first", file.getAbsolutePath());
+ Files.delete(file.toPath());
+ }
+ }
+
+ Optional<LogSegment> logSegmentOption = segments.lastSegment();
+ if (logSegmentOption.isPresent()) logSegmentOption.get().onBecomeInactiveSegment();
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
Review Comment:
Good catch! We can reuse the existing `open` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1461537015
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.function.Supplier;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * A file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * A directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * A directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDirIfExists(dir.toPath());
+ }
+ }
+ }
+
+ /**
+ * Updates end offset of the log, and also updates the recoveryPoint.
+ *
+ * @param endOffset the new end offset of the log
+ */
+ public void updateLogEndOffset(long endOffset) {
+ nextOffsetMetadata = new LogOffsetMetadata(endOffset, segments.activeSegment().baseOffset(), segments.activeSegment().size());
+ if (recoveryPoint > endOffset) {
+ updateRecoveryPoint(endOffset);
+ }
+ }
+
+ /**
+ * Close file handlers used by log but don't write to disk.
+ * This is called if the log directory is offline.
+ */
+ public void closeHandlers() {
+ segments.closeHandlers();
+ isMemoryMappedBufferClosed = true;
+ }
+
+ /**
+ * Closes the segments of the log.
+ */
+ public void close() {
+ maybeHandleIOException(
+ () -> "Error while renaming dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ checkIfMemoryMappedBufferClosed();
+ segments.close();
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete this log directory with no delay.
+ */
+ public void deleteEmptyDir() {
+ maybeHandleIOException(
+ () -> "Error while deleting dir for " + topicPartition + " in dir " + dir.getParent(),
+ (StorageAction<Void, IOException>) () -> {
+ if (segments.nonEmpty()) {
+ throw new IllegalStateException("Can not delete directory when " + segments.numberOfSegments() + " segments are still present");
+ }
+ if (!isMemoryMappedBufferClosed) {
+ throw new IllegalStateException("Can not delete directory when memory mapped buffer for log of " + topicPartition + " is still open.");
+ }
+ Utils.delete(dir);
+ return null;
+ });
+ }
+
+ /**
+ * Completely delete all segments with no delay.
+ *
+ * @return the deleted segments
+ */
+ public Collection<LogSegment> deleteAllSegments() {
+ return maybeHandleIOException(
+ () -> "Error while deleting all segments for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ Collection<LogSegment> allSegments = segments.values();
+ List<LogSegment> deletableSegments = new ArrayList<>(allSegments);
+ removeAndDeleteSegments(allSegments, false, new LogDeletion(this.logger));
+ isMemoryMappedBufferClosed = true;
+ return deletableSegments;
+ });
+ }
+
+ /**
+ * This method deletes the given log segments by doing the following for each of them:
+ * - It removes the segment from the segment map so that it will no longer be used for reads.
+ * - It renames the index and log files by appending .deleted to the respective file name
+ * - It can either schedule an asynchronous delete operation to occur in the future or perform the deletion synchronously
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param segmentsToDelete The log segments to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public void removeAndDeleteSegments(Collection<LogSegment> segmentsToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (!segmentsToDelete.isEmpty()) {
+ // Most callers hold an iterator into the `segments` collection and `removeAndDeleteSegment` mutates it by
+ // removing the deleted segment, we should force materialization of the iterator here, so that results of the
+ // iteration remain valid and deterministic. We should also pass only the materialized view of the
+ // iterator to the logic that actually deletes the segments.
+ List<LogSegment> toDelete = new ArrayList<>(segmentsToDelete);
+ reason.logReason(toDelete);
+ for (LogSegment segment : toDelete) {
+ segments.remove(segment.baseOffset());
+ }
+ deleteSegmentFiles(toDelete, asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+ }
+ }
+
+ /**
+ * This method deletes the given segment and creates a new segment with the given new base offset. It ensures an
+ * active segment exists in the log at all times during this process.
+ * <p>
+ * Asynchronous deletion allows reads to happen concurrently without synchronization and without the possibility of
+ * physically deleting a file while it is being read.
+ * <p>
+ * This method does not convert IOException to KafkaStorageException, the immediate caller
+ * is expected to catch and handle IOException.
+ *
+ * @param newOffset The base offset of the new segment
+ * @param segmentToDelete The old active segment to schedule for deletion
+ * @param asyncDelete Whether the segment files should be deleted asynchronously
+ * @param reason The reason for the segment deletion
+ */
+ public LogSegment createAndDeleteSegment(long newOffset,
+ LogSegment segmentToDelete,
+ boolean asyncDelete,
+ SegmentDeletionReason reason) throws IOException {
+ if (newOffset == segmentToDelete.baseOffset()) {
+ segmentToDelete.changeFileSuffixes("", LogFileUtils.DELETED_FILE_SUFFIX);
+ }
+
+ LogSegment newSegment = LogSegment.open(dir,
+ newOffset,
+ config,
+ time,
+ false,
+ config.initFileSize(),
+ config.preallocate,
+ "");
+ segments.add(newSegment);
+
+ reason.logReason(Collections.singletonList(segmentToDelete));
+ if (newOffset != segmentToDelete.baseOffset()) {
+ segments.remove(segmentToDelete.baseOffset());
+ }
+
+ deleteSegmentFiles(Collections.singletonList(segmentToDelete), asyncDelete, dir, topicPartition, config, scheduler, logDirFailureChannel, logger);
+
+ return newSegment;
+ }
+
+ /**
+ * Given a message offset, find its corresponding offset metadata in the log.
+ * If the message offset is out of range, throw an OffsetOutOfRangeException
+ */
+ public LogOffsetMetadata convertToOffsetMetadataOrThrow(long offset) {
+ FetchDataInfo fetchDataInfo = read(offset, 1, false, nextOffsetMetadata, false);
+ return fetchDataInfo.fetchOffsetMetadata;
+ }
+
+
+ /**
+ * Read messages from the log.
+ *
+ * @param startOffset The offset to begin reading at
+ * @param maxLength The maximum number of bytes to read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxLength` (if one exists)
+ * @param maxOffsetMetadata The metadata of the maximum offset to be fetched
+ * @param includeAbortedTxns If true, aborted transactions are included
+ * @return The fetch data information including fetch starting offset metadata and messages read.
+ * @throws OffsetOutOfRangeException If startOffset is beyond the log end offset
+ */
+ public FetchDataInfo read(long startOffset,
+ int maxLength,
+ boolean minOneMessage,
+ LogOffsetMetadata maxOffsetMetadata,
+ boolean includeAbortedTxns) {
+ return maybeHandleIOException(
+ () -> "Exception while reading for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ logger.trace("Reading maximum {} bytes at offset {} from log with total length {} bytes", maxLength, startOffset, segments.sizeInBytes());
+
+ LogOffsetMetadata endOffsetMetadata = nextOffsetMetadata;
+ long endOffset = endOffsetMetadata.messageOffset;
+ Optional<LogSegment> segmentOpt = segments.floorSegment(startOffset);
+
+ // return error on attempt to read beyond the log end offset
+ if (startOffset > endOffset || !segmentOpt.isPresent())
+ throw new OffsetOutOfRangeException("Received request for offset " + startOffset + " for partition " + topicPartition + ", but we only have log segments upto " + endOffset + ".");
+
+ if (startOffset == maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(maxOffsetMetadata, includeAbortedTxns);
+ else if (startOffset > maxOffsetMetadata.messageOffset)
+ return emptyFetchDataInfo(convertToOffsetMetadataOrThrow(startOffset), includeAbortedTxns);
+ else {
+ // Do the read on the segment with a base offset less than the target offset
+ // but if that segment doesn't contain any messages with an offset greater than that
+ // continue to read from successive segments until we get some messages or we reach the end of the log
+ FetchDataInfo fetchDataInfo = null;
+ while (fetchDataInfo == null && segmentOpt.isPresent()) {
+ LogSegment segment = segmentOpt.get();
+ long baseOffset = segment.baseOffset();
+
+ int maxPosition =
+ // Use the max offset position if it is on this segment; otherwise, the segment size is the limit.
+ maxOffsetMetadata.segmentBaseOffset == segment.baseOffset() ? maxOffsetMetadata.relativePositionInSegment : segment.size();
+
+ fetchDataInfo = segment.read(startOffset, maxLength, maxPosition, minOneMessage);
+ if (fetchDataInfo != null) {
+ if (includeAbortedTxns)
+ fetchDataInfo = addAbortedTransactions(startOffset, segment, fetchDataInfo);
+ } else segmentOpt = segments.higherSegment(baseOffset);
+ }
+
+ if (fetchDataInfo != null) return fetchDataInfo;
+ else {
+ // okay we are beyond the end of the last segment with no data fetched although the start offset is in range,
+ // this can happen when all messages with offset larger than start offsets have been deleted.
+ // In this case, we will return the empty set with log end offset metadata
+ return new FetchDataInfo(nextOffsetMetadata, MemoryRecords.EMPTY);
+ }
+ }
+ });
+ }
+
+ public void append(long lastOffset, long largestTimestamp, long shallowOffsetOfMaxTimestamp, MemoryRecords records) throws IOException {
+ segments.activeSegment().append(lastOffset, largestTimestamp, shallowOffsetOfMaxTimestamp, records);
+ updateLogEndOffset(lastOffset + 1);
+ }
+
+ FetchDataInfo addAbortedTransactions(long startOffset, LogSegment segment, FetchDataInfo fetchInfo) throws IOException {
+ int fetchSize = fetchInfo.records.sizeInBytes();
+ OffsetPosition startOffsetPosition
+ = new OffsetPosition(fetchInfo.fetchOffsetMetadata.messageOffset, fetchInfo.fetchOffsetMetadata.relativePositionInSegment);
+ long upperBoundOffset = segment.fetchUpperBoundOffset(startOffsetPosition, fetchSize)
+ .orElse(segments.higherSegment(segment.baseOffset())
+ .map(LogSegment::baseOffset).orElse(logEndOffset()));
+
+ List<FetchResponseData.AbortedTransaction> abortedTransactions = new ArrayList<>();
+
+ Consumer<List<AbortedTxn>> accumulator = abortedTxns -> abortedTransactions.addAll(abortedTxns.stream()
+ .map(AbortedTxn::asAbortedTransaction).collect(Collectors.toList()));
+
+ collectAbortedTransactions(startOffset, upperBoundOffset, segment, accumulator);
+
+ return new FetchDataInfo(fetchInfo.fetchOffsetMetadata,
+ fetchInfo.records,
+ fetchInfo.firstEntryIncomplete,
+ Optional.of(abortedTransactions));
+ }
+
+ void collectAbortedTransactions(long startOffset,
+ long upperBoundOffset,
+ LogSegment startingSegment,
+ Consumer<List<AbortedTxn>> accumulator) throws IOException {
+ Iterator<LogSegment> higherSegments = segments.higherSegments(startingSegment.baseOffset()).iterator();
+ Optional<LogSegment> segmentEntryOpt = Optional.of(startingSegment);
+ while (segmentEntryOpt.isPresent()) {
+ LogSegment segment = segmentEntryOpt.get();
+ TxnIndexSearchResult searchResult = segment.collectAbortedTxns(startOffset, upperBoundOffset);
+ accumulator.accept(searchResult.abortedTransactions);
+ if (searchResult.isComplete) return;
+ segmentEntryOpt = nextItem(higherSegments);
+ }
+ }
+
+ public List<AbortedTxn> collectAbortedTransactions(long logStartOffset,
+ long baseOffset,
+ long upperBoundOffset) throws IOException {
+ Optional<LogSegment> segmentEntry = segments.floorSegment(baseOffset);
+ List<AbortedTxn> allAbortedTxns = new ArrayList<>();
+
+ if (segmentEntry.isPresent())
+ collectAbortedTransactions(logStartOffset, upperBoundOffset, segmentEntry.get(), allAbortedTxns::addAll);
+
+ return allAbortedTxns;
+ }
+
+ public LogSegment roll() {
+ return roll(OptionalLong.empty());
+ }
+
+ /**
+ * Roll the log over to a new active segment starting with the current logEndOffset.
+ * This will trim the index to the exact size of the number of entries it currently contains.
+ *
+ * @param expectedNextOffset The expected next offset after the segment is rolled
+ * @return The newly rolled segment
+ */
+ public LogSegment roll(OptionalLong expectedNextOffset) {
+ return maybeHandleIOException(
+ () -> "Error while rolling log segment for " + topicPartition + " in dir " + dir.getParent(),
+ () -> {
+ long start = time.hiResClockMs();
+ checkIfMemoryMappedBufferClosed();
+ long newOffset = Math.max(expectedNextOffset.orElse(0L), logEndOffset());
+ File logFile = LogFileUtils.logFile(dir, newOffset, "");
+ LogSegment activeSegment = segments.activeSegment();
+ if (segments.contains(newOffset)) {
+ // segment with the same base offset already exists and loaded
+ if (activeSegment.baseOffset() == newOffset && activeSegment.size() == 0) {
+ // We have seen this happen (see KAFKA-6388) after shouldRoll() returns true for an
+ // active segment of size zero because of one of the indexes is "full" (due to _maxEntries == 0).
Review Comment:
`_maxEntries` is not from `LocalLog` but the `AbstractIndex` class's `maxEntries` field. I updated the doc with `maxEntries`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1440207191
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/SegmentDeletionReason.java:
##########
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.util.List;
+
+public interface SegmentDeletionReason {
Review Comment:
SegmentDeletionReason has other implementations like `RetentionMsBreach`, `RetentionSizeBreach`, `StartOffsetBreach` which use UnifiedLog. We can take a relook at this when we refactor UnifiedLog.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1441,9 +1442,9 @@ class UnifiedLog(@volatile var logStartOffset: Long,
* (if there is one). It returns true iff the segment is deletable.
* @return the segments ready to be deleted
*/
- private[log] def deletableSegments(predicate: (LogSegment, Option[LogSegment]) => Boolean): Iterable[LogSegment] = {
- def isSegmentEligibleForDeletion(nextSegmentOpt: Option[LogSegment], upperBoundOffset: Long): Boolean = {
- val allowDeletionDueToLogStartOffsetIncremented = nextSegmentOpt.isDefined && logStartOffset >= nextSegmentOpt.get.baseOffset
+ private[log] def deletableSegments(predicate: (LogSegment, Optional[LogSegment]) => Boolean): Iterable[LogSegment] = {
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1511,7 +1512,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
localLog.checkIfMemoryMappedBufferClosed()
// remove the segments for lookups
- localLog.removeAndDeleteSegments(segmentsToDelete, asyncDelete = true, reason)
+ localLog.removeAndDeleteSegments(segmentsToDelete.toList.asJava, true, reason)
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
checkstyle/import-control-core.xml:
##########
@@ -37,6 +37,8 @@
<allow pkg="kafka.serializer" />
<allow pkg="org.apache.kafka.common" />
<allow pkg="org.mockito" class="AssignmentsManagerTest"/>
+ <allow pkg="org.apache.kafka.server.util"/>
+ <allow pkg="org.apache.kafka.storage.internals.log"/>
Review Comment:
Right, it is not needed. I guess it was needed when that class was kept inside the core module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1245,7 +1246,7 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
private[log] def collectAbortedTransactions(startOffset: Long, upperBoundOffset: Long): List[AbortedTxn] = {
Review Comment:
I thought these could be addressed when `UnifiedLog` is refactored and moved to storage module.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -2265,11 +2268,12 @@ object UnifiedLog extends Logging {
def deleteProducerSnapshots(): Unit = {
LocalLog.maybeHandleIOException(logDirFailureChannel,
parentDir,
- s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir") {
+ s"Error while deleting producer state snapshots for $topicPartition in dir $parentDir", {
snapshotsToDelete.foreach { snapshot =>
snapshot.deleteIfExists()
}
- }
+ return;
Review Comment:
It was expecting a return for inline declaration, it is throwing a type mismatch error without that.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LocalLog.java:
##########
@@ -0,0 +1,1146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.storage.internals.log;
+
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.errors.KafkaStorageException;
+import org.apache.kafka.common.errors.OffsetOutOfRangeException;
+import org.apache.kafka.common.message.FetchResponseData;
+import org.apache.kafka.common.record.FileLogInputStream;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordVersion;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.util.Scheduler;
+import org.slf4j.Logger;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.StringJoiner;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.BiFunction;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+
+/**
+ * An append-only log for storing messages locally. The log is a sequence of LogSegments, each with a base offset.
+ * New log segments are created according to a configurable policy that controls the size in bytes or time interval
+ * for a given segment.
+ *
+ * NOTE: this class is not thread-safe.
+ */
+public class LocalLog {
+
+ /**
+ * a file that is scheduled to be deleted
+ */
+ public static final String DELETED_FILE_SUFFIX = LogFileUtils.DELETED_FILE_SUFFIX;
+
+ /**
+ * A temporary file that is being used for log cleaning
+ */
+ public static final String CLEANED_FILE_SUFFIX = ".cleaned";
+
+ /**
+ * A temporary file used when swapping files into the log
+ */
+ public static final String SWAP_FILE_SUFFIX = ".swap";
+
+ /**
+ * a directory that is scheduled to be deleted
+ */
+ public static final String DELETE_DIR_SUFFIX = "-delete";
+
+ /**
+ * a directory that is used for future partition
+ */
+ public static final String FUTURE_DIR_SUFFIX = "-future";
+ public static final String STRAY_DIR_SUFFIX = "-stray";
+
+ public static final Pattern DELETE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + DELETE_DIR_SUFFIX);
+ public static final Pattern FUTURE_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + FUTURE_DIR_SUFFIX);
+ public static final Pattern STRAY_DIR_PATTERN = Pattern.compile("^(\\S+)-(\\S+)\\.(\\S+)" + STRAY_DIR_SUFFIX);
+
+ public static final long UNKNOWN_OFFSET = -1L;
+
+ private final Logger logger;
+
+ private final LogSegments segments;
+ private final Scheduler scheduler;
+ private final Time time;
+ private final TopicPartition topicPartition;
+ private final LogDirFailureChannel logDirFailureChannel;
+
+ // Last time the log was flushed
+ private final AtomicLong lastFlushedTime;
+
+ private volatile File dir;
+ private volatile LogConfig config;
+ private volatile long recoveryPoint;
+ private volatile LogOffsetMetadata nextOffsetMetadata;
+
+ // Cache value of parent directory to avoid allocations in hot paths like ReplicaManager.checkpointHighWatermarks
+ private volatile String parentDir;
+ // The memory mapped buffer for index files of this log will be closed with either delete() or closeHandlers()
+ // After memory mapped buffer is closed, no disk IO operation should be performed for this log.
+ private volatile boolean isMemoryMappedBufferClosed = false;
+
+ /**
+ * Creates a new LocalLog instance.
+ *
+ * @param dir The directory in which log segments are created.
+ * @param config The log configuration settings
+ * @param segments The non-empty log segments recovered from disk
+ * @param recoveryPoint The offset at which to begin the next recovery i.e. the first offset which has not been flushed to disk
+ * @param scheduler The thread pool scheduler used for background actions
+ * @param nextOffsetMetadata The offset where the next message could be appended
+ * @param time The time instance used for checking the clock
+ * @param topicPartition The topic partition associated with this log
+ * @param logDirFailureChannel The LogDirFailureChannel instance to asynchronously handle Log dir failure
+ */
+ public LocalLog(File dir,
+ LogConfig config,
+ LogSegments segments,
+ long recoveryPoint,
+ LogOffsetMetadata nextOffsetMetadata,
+ Scheduler scheduler,
+ Time time,
+ TopicPartition topicPartition,
+ LogDirFailureChannel logDirFailureChannel) {
+ this.dir = dir;
+ this.config = config;
+ this.segments = segments;
+ this.recoveryPoint = recoveryPoint;
+ this.scheduler = scheduler;
+ this.nextOffsetMetadata = nextOffsetMetadata;
+ this.time = time;
+ this.topicPartition = topicPartition;
+ this.logDirFailureChannel = logDirFailureChannel;
+
+ parentDir = dir.getParent();
+ lastFlushedTime = new AtomicLong(time.milliseconds());
+ String logPrefix = "[LocalLog partition=" + topicPartition + ", dir=" + dir + "] ";
+ logger = new LogContext(logPrefix).logger(LocalLog.class);
+ }
+
+ public Logger logger() {
+ return logger;
+ }
+
+ public LogConfig config() {
+ return config;
+ }
+
+ public LogSegments segments() {
+ return segments;
+ }
+
+ public Scheduler scheduler() {
+ return scheduler;
+ }
+
+ public LogOffsetMetadata nextOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ public TopicPartition topicPartition() {
+ return topicPartition;
+ }
+
+ public LogDirFailureChannel logDirFailureChannel() {
+ return logDirFailureChannel;
+ }
+
+ public long recoveryPoint() {
+ return recoveryPoint;
+ }
+
+ public Time time() {
+ return time;
+ }
+
+ /**
+ * The time this log is last known to have been fully flushed to disk
+ */
+ public long lastFlushTime() {
+ return lastFlushedTime.get();
+ }
+
+ /**
+ * The offset metadata of the next message that will be appended to the log
+ */
+ public LogOffsetMetadata logEndOffsetMetadata() {
+ return nextOffsetMetadata;
+ }
+
+ /**
+ * The offset of the next message that will be appended to the log
+ */
+ public long logEndOffset() {
+ return nextOffsetMetadata.messageOffset;
+ }
+
+ public File dir() {
+ return dir;
+ }
+
+ public String name() {
+ return dir.getName();
+ }
+
+ public String parentDir() {
+ return parentDir;
+ }
+
+ public File parentDirFile() {
+ return new File(parentDir);
+ }
+
+ public boolean isFuture() {
+ return dir.getName().endsWith(LocalLog.FUTURE_DIR_SUFFIX);
+ }
+
+ /**
+ * Rename the directory of the local-log to the given name.
+ *
+ * @param name the new dir name
+ * @throws KafkaStorageException if rename fails
+ */
+ public boolean renameDir(String name) {
+ return maybeHandleIOException("Error while renaming dir for " + topicPartition + " in log dir " + dir.getParent(),
+ () -> {
+ File renamedDir = new File(dir.getParent(), name);
+ Utils.atomicMoveWithFallback(dir.toPath(), renamedDir.toPath());
+ if (!renamedDir.equals(dir)) {
+ dir = renamedDir;
+ parentDir = renamedDir.getParent();
+ segments.updateParentDir(renamedDir);
+ return true;
+ } else {
+ return false;
+ }
+ });
+ }
+
+ /**
+ * Update the existing configuration to the new provided configuration.
+ *
+ * @param newConfig the new configuration to be updated to
+ */
+ public void updateConfig(LogConfig newConfig) {
+ LogConfig oldConfig = config;
+ config = newConfig;
+ RecordVersion oldRecordVersion = oldConfig.recordVersion();
+ RecordVersion newRecordVersion = newConfig.recordVersion();
+ if (newRecordVersion.precedes(oldRecordVersion))
+ logger.warn("Record format version has been downgraded from {} to {}.", oldRecordVersion, newRecordVersion);
+ }
+
+ public void checkIfMemoryMappedBufferClosed() {
+ if (isMemoryMappedBufferClosed)
+ throw new KafkaStorageException("The memory mapped buffer for log of " + topicPartition + " is already closed");
+ }
+
+ public void updateRecoveryPoint(long newRecoveryPoint) {
+ recoveryPoint = newRecoveryPoint;
+ }
+
+ /**
+ * Update recoveryPoint to provided offset and mark the log as flushed, if the offset is greater
+ * than the existing recoveryPoint.
+ *
+ * @param offset the offset to be updated
+ */
+ public void markFlushed(long offset) {
+ checkIfMemoryMappedBufferClosed();
+ if (offset > recoveryPoint) {
+ updateRecoveryPoint(offset);
+ lastFlushedTime.set(time.milliseconds());
+ }
+ }
+
+ /**
+ * The number of messages appended to the log since the last flush
+ */
+ public long unflushedMessages() {
+ return logEndOffset() - recoveryPoint;
+ }
+
+ /**
+ * Flush local log segments for all offsets up to offset-1.
+ * Note: It does not update the recovery point.
+ *
+ * @param offset The offset to flush up to (non-inclusive)
+ */
+ public void flush(long offset) throws IOException {
+ long currentRecoveryPoint = recoveryPoint;
+ if (currentRecoveryPoint <= offset) {
+ Collection<LogSegment> segmentsToFlush = segments.values(currentRecoveryPoint, offset);
+ for (LogSegment segment : segmentsToFlush) {
+ segment.flush();
+ }
+ // If there are any new segments, we need to flush the parent directory for crash consistency.
+ if (segmentsToFlush.stream().anyMatch(x -> x.baseOffset() >= currentRecoveryPoint)) {
+ // The directory might be renamed concurrently for topic deletion, which may cause NoSuchFileException here.
+ // Since the directory is to be deleted anyways, we just swallow NoSuchFileException and let it go.
+ Utils.flushDir(dir.toPath());
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14034:
URL: https://github.com/apache/kafka/pull/14034#discussion_r1440207191
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/SegmentDeletionReason.java:
##########
@@ -0,0 +1,23 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.util.List;
+
+public interface SegmentDeletionReason {
Review Comment:
SegmentDeletionReason has other implementations like `RetentionMsBreach`, `RetentionSizeBreach`, `StartOffsetBreach` which use UnifiedLog. We can take a relook at this when we refactor UnifiedLog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14483 Move/Rewrite of LocalLog to storage module. [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd closed pull request #14034: KAFKA-14483 Move/Rewrite of LocalLog to storage module.
URL: https://github.com/apache/kafka/pull/14034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org