You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "ijuma (via GitHub)" <gi...@apache.org> on 2023/10/11 13:54:31 UTC
[PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
ijuma opened a new pull request, #14529:
URL: https://github.com/apache/kafka/pull/14529
For broader context on this change, please check:
* KAFKA-14470: Move log layer to storage module
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1360163730
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,889 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment implements Closeable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /**
+ * The maximum timestamp we see so far.
+ *
+ * Note that this may result in time index materialization.
+ */
+ public long maxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().timestamp;
+ }
+
+ /**
+ * Note that this may result in time index materialization.
+ */
+ private long offsetOfMaxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ private boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ // We don't call `largestRecordTimestamp` below to avoid materializing the time index when `toString` is invoked
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + maxTimestampAndOffsetSoFar.timestamp +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
+ }
+ }
+
+ /**
+ * Update the directory reference for the log and indices in this segment. This would typically be called after a
+ * directory is renamed.
+ */
+ void updateParentDir(File dir) {
+ log.updateParentDir(dir);
+ lazyOffsetIndex.updateParentDir(dir);
+ lazyTimeIndex.updateParentDir(dir);
+ txnIndex.updateParentDir(dir);
+ }
+
+ /**
+ * Change the suffix for the index and log files for this log segment
+ * IOException from this method should be handled by the caller
+ */
+ public void changeFileSuffixes(String oldSuffix, String newSuffix) throws IOException {
+ log.renameTo(new File(Utils.replaceSuffix(log.file().getPath(), oldSuffix, newSuffix)));
+ lazyOffsetIndex.renameTo(new File(Utils.replaceSuffix(offsetIndexFile().getPath(), oldSuffix, newSuffix)));
+ lazyTimeIndex.renameTo(new File(Utils.replaceSuffix(timeIndexFile().getPath(), oldSuffix, newSuffix)));
+ txnIndex.renameTo(new File(Utils.replaceSuffix(txnIndex.file().getPath(), oldSuffix, newSuffix)));
+ }
+
+ public boolean hasSuffix(String suffix) {
+ return log.file().getName().endsWith(suffix) &&
+ offsetIndexFile().getName().endsWith(suffix) &&
+ timeIndexFile().getName().endsWith(suffix) &&
+ txnIndex.file().getName().endsWith(suffix);
+ }
+
+ /**
+ * Append the largest time index entry to the time index and trim the log and indexes.
+ *
+ * The time index entry appended will be used to decide when to delete the segment.
+ */
+ public void onBecomeInactiveSegment() throws IOException {
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ offsetIndex().trimToValidSize();
+ timeIndex().trimToValidSize();
+ log.trim();
+ }
+
+ /**
+ * If not previously loaded,
+ * load the timestamp of the first message into memory.
+ */
+ private void loadFirstBatchTimestamp() {
+ if (!rollingBasedTimestamp.isPresent()) {
+ Iterator<FileChannelRecordBatch> iter = log.batches().iterator();
+ if (iter.hasNext())
+ rollingBasedTimestamp = OptionalLong.of(iter.next().maxTimestamp());
+ }
+ }
+
+ /**
+ * The time this segment has waited to be rolled.
+ * If the first message batch has a timestamp we use its timestamp to determine when to roll a segment. A segment
+ * is rolled if the difference between the new batch's timestamp and the first batch's timestamp exceeds the
+ * segment rolling time.
+ * If the first batch does not have a timestamp, we use the wall clock time to determine when to roll a segment. A
+ * segment is rolled if the difference between the current wall clock time and the segment create time exceeds the
+ * segment rolling time.
+ */
+ public long timeWaitedForRoll(long now, long messageTimestamp) {
+ // Load the timestamp of the first message into memory
+ loadFirstBatchTimestamp();
+ long ts = rollingBasedTimestamp.orElse(-1L);
+ if (ts >= 0)
+ return messageTimestamp - ts;
+ return now - created;
+ }
+
+ /**
+ * @return the first batch timestamp if the timestamp is available. Otherwise return Long.MaxValue
+ */
+ long getFirstBatchTimestamp() {
+ loadFirstBatchTimestamp();
+ OptionalLong timestamp = rollingBasedTimestamp;
+ if (timestamp.isPresent() && timestamp.getAsLong() >= 0)
+ return timestamp.getAsLong();
+ return Long.MAX_VALUE;
+ }
+
+ /**
+ * Search the message offset based on timestamp and offset.
+ *
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ *
+ * - If all the messages in the segment have smaller offsets, return None
+ * - If all the messages in the segment have smaller timestamps, return None
+ * - If all the messages in the segment have larger timestamps, or no message in the segment has a timestamp
+ * the returned the offset will be max(the base offset of the segment, startingOffset) and the timestamp will be Message.NoTimestamp.
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * This method only returns None when 1) all messages' offset < startOffing or 2) the log is not empty but we did not
+ * see any message when scanning the log from the indexed position. The latter could happen if the log is truncated
+ * after we get the indexed position but before we scan the log from there. In this case we simply return None and the
+ * caller will need to check on the truncated log and maybe retry or even do the search on another log segment.
+ *
+ * @param timestampMs The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(long timestampMs, long startingOffset) throws IOException {
+ // Get the index entry with a timestamp less than or equal to the target timestamp
+ TimestampOffset timestampOffset = timeIndex().lookup(timestampMs);
+ int position = offsetIndex().lookup(Math.max(timestampOffset.offset, startingOffset)).position;
+
+ // Search the timestamp
+ return Optional.ofNullable(log.searchForTimestamp(timestampMs, position, startingOffset));
+ }
+
+ /**
+ * Close this log segment
+ */
+ @Override
+ public void close() throws IOException {
+ if (maxTimestampAndOffsetSoFar != TimestampOffset.UNKNOWN)
+ Utils.swallow(LOGGER, Level.WARN, "maybeAppend", () -> timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true));
+ Utils.closeQuietly(lazyOffsetIndex, "offsetIndex", LOGGER);
Review Comment:
Nice to use `closeQuietly` instead of using swallow by closing here.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,889 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment implements Closeable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /**
+ * The maximum timestamp we see so far.
+ *
+ * Note that this may result in time index materialization.
+ */
+ public long maxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().timestamp;
+ }
+
+ /**
+ * Note that this may result in time index materialization.
+ */
+ private long offsetOfMaxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ private boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ // We don't call `largestRecordTimestamp` below to avoid materializing the time index when `toString` is invoked
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + maxTimestampAndOffsetSoFar.timestamp +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
Review Comment:
Good change to maintain the semantics while moving to Java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358278974
##########
core/src/main/scala/kafka/utils/CoreUtils.scala:
##########
@@ -66,11 +66,11 @@ object CoreUtils {
action
} catch {
case e: Throwable => logLevel match {
- case Level.ERROR => logger.error(e.getMessage, e)
- case Level.WARN => logger.warn(e.getMessage, e)
- case Level.INFO => logger.info(e.getMessage, e)
- case Level.DEBUG => logger.debug(e.getMessage, e)
- case Level.TRACE => logger.trace(e.getMessage, e)
+ case Level.ERROR => logging.error(e.getMessage, e)
+ case Level.WARN => logging.warn(e.getMessage, e)
+ case Level.INFO => logging.info(e.getMessage, e)
+ case Level.DEBUG => logging.debug(e.getMessage, e)
+ case Level.TRACE => logging.trace(e.getMessage, e)
Review Comment:
I vote for separate JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358317031
##########
core/src/main/scala/kafka/utils/CoreUtils.scala:
##########
@@ -66,11 +66,11 @@ object CoreUtils {
action
} catch {
case e: Throwable => logLevel match {
- case Level.ERROR => logger.error(e.getMessage, e)
- case Level.WARN => logger.warn(e.getMessage, e)
- case Level.INFO => logger.info(e.getMessage, e)
- case Level.DEBUG => logger.debug(e.getMessage, e)
- case Level.TRACE => logger.trace(e.getMessage, e)
+ case Level.ERROR => logging.error(e.getMessage, e)
+ case Level.WARN => logging.warn(e.getMessage, e)
+ case Level.INFO => logging.info(e.getMessage, e)
+ case Level.DEBUG => logging.debug(e.getMessage, e)
+ case Level.TRACE => logging.trace(e.getMessage, e)
Review Comment:
separate JIRA is good. Could be a great newbie task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356966266
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
Review Comment:
`group` and `type` is referring to the parameters we are passing to `explicitMetricName`, so I think it's explicit. I tweaked the comment a bit more to hopefully make it clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358301179
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
Review Comment:
I did. There were exactly 3 cases, the ones you pointed out. You don't need to sell me on the benefit, I filed KAFKA-14489 from the start of this effort. ;)
I originally intended to do the `LogSegments` conversion in a separate PR and the 3 examples couldn't be default package at first. Move once I added the `LogSegments` conversion to this PR, it opened up that opportunity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356886567
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
Review Comment:
No, `UnifiedLog` is in a different package right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356883718
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
It's a public because it needs to be a public method. What you suggested doesn't work because it forces materialization and it would be a serious regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1357965208
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
Review Comment:
@ijuma thoughts on this? You can choose to use Closeable too if you like it better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1357977959
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
We are leaking the fact here that the index is implemented using a lazy index implementation outside the segment.
Does the caller know that File returned by `timeIndexFile()` may not be consistent with the in-memory state of index? We are adding the responsibility on the caller to ensure that it calls flush() if it requires consistency. This means that we are leaking the internal implementation of index (being lazy) to the caller. This is concerning because it can cause bugs where authors using this method in other parts of the code may not understand that it could be eventually inconsistent.
If we really want to provide an index reference which doesn't require materialization, why not share the LazyIndex with the caller. Then the caller explicitly knows that this index is lazily evaluated?
Also, where are we using this function outside this class? Should this be private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1360877868
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -733,8 +733,8 @@ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegment
remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
- LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
- toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.offsetIndex().file()),
+ toPathIfExists(segment.timeIndex().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
Review Comment:
@ijuma, Good catch! afaik, no need to load the lazyIndex to write the contents of the index files to remote storage.
@divij Right, we do not need to materialize the indexes for RSM to write them to the remote storage. Whenever a log segment is rolled over, segment and its indexes are flushed to disk in an asynchrnous manner. As indexes are mmapped, any file reads fetch from page cache which will be consistent with whatever is written to memory. We can explore whether it is really needed to flush it. RLM has access to segment, that can be flushed before the files are passed to RSM.
Filed https://issues.apache.org/jira/browse/KAFKA-15612 for follow-up discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356881661
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java:
##########
@@ -41,7 +41,7 @@
* Methods of this class are thread safe. Make sure to check `AbstractIndex` subclasses
* documentation to establish their thread safety.
*/
-public class LazyIndex<T extends AbstractIndex> {
+public class LazyIndex<T extends AbstractIndex> implements Closeable {
Review Comment:
Why is that better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1760584199
Test failures:
> Build / JDK 17 and Scala 2.13 / kafka.api.PlaintextConsumerTest.testSubsequentPatternSubscription()
Build / JDK 21 and Scala 2.13 / kafka.api.AuthorizerIntegrationTest.testAuthorizeByResourceTypePrefixedResourceDenyDominate(String).quorum=zk
Build / JDK 21 and Scala 2.13 / kafka.api.ConsumerBounceTest.testCloseDuringRebalance()
Build / JDK 21 and Scala 2.13 / kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.testCreateTokenForOtherUserFails(String).quorum=kraft
Build / JDK 21 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()
Build / JDK 11 and Scala 2.13 / integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor()
Build / JDK 11 and Scala 2.13 / org.apache.kafka.tools.MetadataQuorumCommandTest.[1] Type=Raft-Combined, Name=testDescribeQuorumReplicationSuccessful, MetadataVersion=3.7-IV0, Security=PLAINTEXT
Build / JDK 11 and Scala 2.13 / org.apache.kafka.tools.MetadataQuorumCommandTest.[1] Type=Raft-Combined, Name=testDescribeQuorumReplicationSuccessful, MetadataVersion=3.7-IV0, Security=PLAINTEXT
They look unrelated, but I kicked off another build to get more signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356629072
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -733,8 +733,8 @@ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegment
remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
- LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
- toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.offsetIndex().file()),
+ toPathIfExists(segment.timeIndex().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
Review Comment:
This metadata is passed to RSM plugin which is external to Kafka. I would like to hide the details (that we have a lazy materialized index) from the external RSM plugin and instead have a clean contract which states, "Kafka guarantees that these files will be present, RSM can pick up and upload these". This provides a clean decoupling where Kafka <-> RSM plugin state sharing is only via files.
That is why we need to materialize the indexes before giving them to RSM plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358023201
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
Review Comment:
Could we check for functions used only by `LogSegments`? I left a comment for few of them as examples. I am insisting on this because having a tighter/strict contract even between internal code reduces probability of bugs since implementation (and other details such as lifecycle) remains encapsulated to a component. Then, other callers can call public functions without having to worry about gotchas or unexpected side-effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358592259
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
The indexes are memory mapped, so in theory it should be consistent for reads that go through the page cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358569570
##########
clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java:
##########
@@ -44,6 +45,17 @@ public RecordBatch firstBatch() {
return iterator.next();
}
+ @Override
+ public Optional<RecordBatch> lastBatch() {
+ Iterator<? extends RecordBatch> iterator = batches().iterator();
+
+ RecordBatch batch = null;
+ while (iterator.hasNext())
+ batch = iterator.next();
+
+ return Optional.ofNullable(batch);
Review Comment:
Never mind. My misunderstanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1357961584
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
ok, I will leave it up to you.
In case you are curious, my thoughts were as follows but this is a trivial point and we can agree to disagree.
Do we anticipate concrete implementations of `Records` to provide an optimized implementation of `firstBatch()` and `lastBatch()` compared to scanning using batchIterator()? If yes, then they should be added to the contract. If we are only adding it here to hide the verbosity of same implementation that a callee will do, then should just use it as a helper (and not an interface methods). Helper methods go in abstract base class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356893064
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + largestRecordTimestamp() +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
Review Comment:
Not really, this is a change to make the compiler happy, but we never throw a checked exception that is not an IOException. If the code ever changes to do that, we'll catch it here. I think it's fine as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356883718
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
It's public because it needs to be a public method (this is an internal class though). What you suggested doesn't work because it forces materialization and it would be a serious regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358343260
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
Review Comment:
> I filed [KAFKA-14489](https://issues.apache.org/jira/browse/KAFKA-14489) from the start of this effort.
Nice! I thought my comment needed explanation because you pushed back on the index implementation leak discussion in the other comment. Glad to see that we are on same page on restricting access amongst components (as long as it doesn't impact perf such as index case).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1355277747
##########
core/src/main/scala/kafka/utils/CoreUtils.scala:
##########
@@ -66,11 +66,11 @@ object CoreUtils {
action
} catch {
case e: Throwable => logLevel match {
- case Level.ERROR => logger.error(e.getMessage, e)
- case Level.WARN => logger.warn(e.getMessage, e)
- case Level.INFO => logger.info(e.getMessage, e)
- case Level.DEBUG => logger.debug(e.getMessage, e)
- case Level.TRACE => logger.trace(e.getMessage, e)
+ case Level.ERROR => logging.error(e.getMessage, e)
+ case Level.WARN => logging.warn(e.getMessage, e)
+ case Level.INFO => logging.info(e.getMessage, e)
+ case Level.DEBUG => logging.debug(e.getMessage, e)
+ case Level.TRACE => logging.trace(e.getMessage, e)
Review Comment:
Fixed a bug where we were not using the passed `logging`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1761166784
@ijuma Thanks for the PR. I will review it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358373579
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
`LazyIndex` exposes a larger surface area though. And it's confusing to expose both `LazyIndex` and the materialized index. The way it is now is actually simpler. Local indexes are file based and that's core to how they work. `LogSegment` gives you the file if that's all you need or the materialized index. And `LazyIndex` is used internally to simplify the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358282888
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
Review Comment:
It seems unrelated to this change (I needed to do it for `LazyIndex` in order to use `closeQuietly`), but I'll include it my next push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356970370
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
Review Comment:
I added a `read` prefix to this to make it clearer that it does more than than the field `maxTimestampAndOffsetSoFar`. The conversion from Scala to Java had originally caused some code to use the field instead of the method, which lead to some subtly different behavior in some cases.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
Review Comment:
I added a `read` prefix to this to make it clearer that it does more than than the field `maxTimestampAndOffsetSoFar`. The conversion from Scala to Java had originally caused some code to use the field instead of the method, which led to some subtly different behavior in some cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358023791
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1360684748
##########
core/src/main/scala/kafka/utils/CoreUtils.scala:
##########
@@ -66,11 +66,11 @@ object CoreUtils {
action
} catch {
case e: Throwable => logLevel match {
- case Level.ERROR => logger.error(e.getMessage, e)
- case Level.WARN => logger.warn(e.getMessage, e)
- case Level.INFO => logger.info(e.getMessage, e)
- case Level.DEBUG => logger.debug(e.getMessage, e)
- case Level.TRACE => logger.trace(e.getMessage, e)
+ case Level.ERROR => logging.error(e.getMessage, e)
+ case Level.WARN => logging.warn(e.getMessage, e)
+ case Level.INFO => logging.info(e.getMessage, e)
+ case Level.DEBUG => logging.debug(e.getMessage, e)
+ case Level.TRACE => logging.trace(e.getMessage, e)
Review Comment:
https://issues.apache.org/jira/browse/KAFKA-15610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1759698972
Thanks for the prompt review @divijvaidya! Please note that this is still in draft because there are still failing tests. Since it was still in draft mode, I have been force pushing and so on. In the future, please let me know if you are reviewing a draft PR so I can avoid force pushes (which are painful for you as the reviewer).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356961289
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -120,12 +120,13 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
// Set the last modified time to an old value to force deletion of old segments
val endOffset = log.logEndOffset
- log.logSegments.foreach(_.lastModified = time.milliseconds - (2 * retentionMs))
+ log.logSegments.forEach(_.setLastModified(time.milliseconds - (2 * retentionMs)))
TestUtils.waitUntilTrue(() => log.logStartOffset == endOffset,
"Timed out waiting for deletion of old segments")
assertEquals(1, log.numberOfSegments)
cleaner.shutdown()
+ closeLog(log)
Review Comment:
I noticed we did not close the log before reopening it a bit later. It didn't seem to cause any problems, but it made it more difficult to debug test failures (since the behavior is not clearly defined in this case).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358363210
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
> We are not leaking anything that is not already exposed. It would be different if that was not accessible via the overall interface already. The only discussion here is not whether this is exposed (it already is), but how to expose it.
Fair point. Let tackle the how question in this PR and take up plugging the underlying file implementation leak separately later.
On the how part, can we keep the index or LaxyIndex as the components who choose to expose internal implementation outside (instead of LogSegment)? What that means is that we will probably have a function here in LogSegment which returns LazyIndex.
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1761985192
> Thank you for patiently answering my comments
Thanks for the review and for paying close attention to code quality (modularity, readability, etc.) - it's an important area!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358285270
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
We are not leaking anything that is not already exposed. It would be different if that was not accessible via the overall interface already. The only discussion here is not whether this is exposed (it already is), but _how_ to expose it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1360649801
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -733,8 +733,8 @@ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegment
remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
- LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
- toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.offsetIndex().file()),
+ toPathIfExists(segment.timeIndex().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
Review Comment:
Perhaps we can file the JIRA and discuss it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358306908
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
Review Comment:
I'm not sure what to add as a comment, static classes basically last for the life of the JVM. I would personally prefer to have this metric be scoped to the life of a kafka server, but that would be a behavior change and best handled via a separate JIRA/PR. I simply preserved the behavior here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358666639
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
One last clarification: I am not opposed to improving the overall modularity of the classes in the storage layer. As you said, in a separate PR/discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356896786
##########
clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java:
##########
@@ -44,6 +45,17 @@ public RecordBatch firstBatch() {
return iterator.next();
}
+ @Override
+ public Optional<RecordBatch> lastBatch() {
+ Iterator<? extends RecordBatch> iterator = batches().iterator();
+
+ RecordBatch batch = null;
+ while (iterator.hasNext())
+ batch = iterator.next();
+
+ return Optional.ofNullable(batch);
Review Comment:
I don't follow the question. `Optional.ofNullable` converts `null` to `Optional.empty`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356912887
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java:
##########
@@ -41,7 +41,7 @@
* Methods of this class are thread safe. Make sure to check `AbstractIndex` subclasses
* documentation to establish their thread safety.
*/
-public class LazyIndex<T extends AbstractIndex> {
+public class LazyIndex<T extends AbstractIndex> implements Closeable {
Review Comment:
`AutoCloseable` can be used with try-with-resource in Java whereas Closeable cannot.
[1] https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356889538
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
Review Comment:
It cannot since `UnifiedLog` is in a different package right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356970370
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
Review Comment:
I added a `read` prefix to this to make it clearer that it does more than `maxTimestampAndOffsetSoFar`. The conversion from Scala to Java had originally caused some code to use the field instead of the method, which lead to some subtly different behavior in some cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1759952077
The Java 11 build has 3 unrelated failures:
> Build / JDK 11 and Scala 2.13 / org.apache.kafka.common.network.SslTransportLayerTest.[3] tlsProtocol=TLSv1.3, useInlinePem=false
Build / JDK 11 and Scala 2.13 / org.apache.kafka.streams.integration.NamedTopologyIntegrationTest.shouldAddNamedTopologyToRunningApplicationWithSingleInitialNamedTopology()
Build / JDK 11 and Scala 2.13 / org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest.testFallbackPriorTaskAssignorLargePartitionCount
This is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358302231
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
`AbstractRecords` mostly exists because default methods weren't a thing then. Abstract classes that don't add any value like this one are a bit of anti-pattern (since the abstract class is public too, it doesn't encapsulate anything).
You can see the evolution of the standard library to add default methods to various interfaces in recent times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358341201
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
Interesting. Thanks. TIL!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358566629
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
ok, could we add a javadoc on this method that the physical file pointed to by this file object may not be consistent with in-memory copy of index? It doesn't totally address my concern of a potential bug when someone uses this File and assumes that it represents a consistent view of the index but I will am happy with a javadoc now. The whole index implemented by a file thing needs to be hidden away but that belongs in a separate JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358302231
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
`AbstractRecords` only exists because default methods weren't a thing then. Abstract classes that don't add any value like this one are a bit of anti-pattern (since the abstract class is public too, it doesn't encapsulate anything).
You can see the evolution of the standard library to add default methods to various interfaces in recent times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358364613
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
Review Comment:
ok, we can leave it as is it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1360458809
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -733,8 +733,8 @@ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegment
remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
- LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
- toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.offsetIndex().file()),
+ toPathIfExists(segment.timeIndex().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
Review Comment:
I have been thinking about this. I take back my original statement.
We want to ensure that the file we are passing to RSM plugin contains all the data which is present in MemoryByteBuffer i.e. we should have flushed the MemoryByteBuffer to the file using force(). In Kafka, when we close a segment, indexes are flushed asynchronously. Hence, it might be possible that when we are passing the file to RSM, the file doesn't contain flushed data. This is a bug but it is not related to the change we are trying to make here. I will create a separate JIRA for this.
RSM doesn't need to read the index file during archive, hence it's ok to pass just the file name "without" materializing the index in the in-memory mmapped buffers.
Hence, we should flush() the content of indexes into the file before this operation but it is not necessary to materialize (i.e. read the content of file into memory).
@satishd we require your opinion here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma merged PR #14529:
URL: https://github.com/apache/kafka/pull/14529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1355263700
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -733,8 +733,8 @@ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegment
remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
- LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
- toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.offsetIndex().file()),
+ toPathIfExists(segment.timeIndex().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
Review Comment:
@satishd Is it intentional that we force the indexes to be materialized here? We could pass the file without materializing if that's better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358020016
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + largestRecordTimestamp() +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358306908
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
Review Comment:
I'm not sure what to add as a comment, static fields basically last for the life of the JVM. I would personally prefer to have this metric be scoped to the life of a kafka server, but that would be a behavior change and best handled via a separate JIRA/PR. I simply preserved the behavior here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356568343
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java:
##########
@@ -41,7 +41,7 @@
* Methods of this class are thread safe. Make sure to check `AbstractIndex` subclasses
* documentation to establish their thread safety.
*/
-public class LazyIndex<T extends AbstractIndex> {
+public class LazyIndex<T extends AbstractIndex> implements Closeable {
Review Comment:
`AutoCloseable`?
Also please add `Override` annotation on the close() method.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
We are leaking the implementation detail of index here (the fact that it is backed by a file). IMO, this should not be a public method. If someone wants to access the underlying index file, they should be asking the Index about it and not the LogSegment using LogSegment.timeIndex().file() instead.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
Review Comment:
This should be private. Only this class uses it.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + largestRecordTimestamp() +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
+ }
+ }
+
+ /**
+ * Update the directory reference for the log and indices in this segment. This would typically be called after a
+ * directory is renamed.
+ */
+ public void updateParentDir(File dir) {
+ log.updateParentDir(dir);
+ lazyOffsetIndex.updateParentDir(dir);
+ lazyTimeIndex.updateParentDir(dir);
+ txnIndex.updateParentDir(dir);
+ }
+
+ /**
+ * Change the suffix for the index and log files for this log segment
+ * IOException from this method should be handled by the caller
+ */
+ public void changeFileSuffixes(String oldSuffix, String newSuffix) throws IOException {
+ log.renameTo(new File(Utils.replaceSuffix(log.file().getPath(), oldSuffix, newSuffix)));
+ lazyOffsetIndex.renameTo(new File(Utils.replaceSuffix(offsetIndexFile().getPath(), oldSuffix, newSuffix)));
+ lazyTimeIndex.renameTo(new File(Utils.replaceSuffix(timeIndexFile().getPath(), oldSuffix, newSuffix)));
+ txnIndex.renameTo(new File(Utils.replaceSuffix(txnIndex.file().getPath(), oldSuffix, newSuffix)));
+ }
+
+ public boolean hasSuffix(String suffix) {
+ return log.file().getName().endsWith(suffix) &&
+ offsetIndexFile().getName().endsWith(suffix) &&
+ timeIndexFile().getName().endsWith(suffix) &&
+ txnIndex.file().getName().endsWith(suffix);
+ }
+
+ /**
+ * Append the largest time index entry to the time index and trim the log and indexes.
+ *
+ * The time index entry appended will be used to decide when to delete the segment.
+ */
+ public void onBecomeInactiveSegment() throws IOException {
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ offsetIndex().trimToValidSize();
+ timeIndex().trimToValidSize();
+ log.trim();
+ }
+
+ /**
+ * If not previously loaded,
+ * load the timestamp of the first message into memory.
+ */
+ private void loadFirstBatchTimestamp() {
+ if (!rollingBasedTimestamp.isPresent()) {
+ Iterator<FileChannelRecordBatch> iter = log.batches().iterator();
+ if (iter.hasNext())
+ rollingBasedTimestamp = OptionalLong.of(iter.next().maxTimestamp());
+ }
+ }
+
+ /**
+ * The time this segment has waited to be rolled.
+ * If the first message batch has a timestamp we use its timestamp to determine when to roll a segment. A segment
+ * is rolled if the difference between the new batch's timestamp and the first batch's timestamp exceeds the
+ * segment rolling time.
+ * If the first batch does not have a timestamp, we use the wall clock time to determine when to roll a segment. A
+ * segment is rolled if the difference between the current wall clock time and the segment create time exceeds the
+ * segment rolling time.
+ */
+ public long timeWaitedForRoll(long now, long messageTimestamp) {
+ // Load the timestamp of the first message into memory
+ loadFirstBatchTimestamp();
+ long ts = rollingBasedTimestamp.orElse(-1L);
+ if (ts >= 0)
+ return messageTimestamp - ts;
+ return now - created;
+ }
+
+ /**
+ * @return the first batch timestamp if the timestamp is available. Otherwise return Long.MaxValue
+ */
+ public long getFirstBatchTimestamp() {
+ loadFirstBatchTimestamp();
+ OptionalLong timestamp = rollingBasedTimestamp;
+ if (timestamp.isPresent() && timestamp.getAsLong() >= 0)
+ return timestamp.getAsLong();
+ return Long.MAX_VALUE;
+ }
+
+ /**
+ * Search the message offset based on timestamp and offset.
+ *
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ *
+ * - If all the messages in the segment have smaller offsets, return None
+ * - If all the messages in the segment have smaller timestamps, return None
+ * - If all the messages in the segment have larger timestamps, or no message in the segment has a timestamp
+ * the returned the offset will be max(the base offset of the segment, startingOffset) and the timestamp will be Message.NoTimestamp.
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * This method only returns None when 1) all messages' offset < startOffing or 2) the log is not empty but we did not
+ * see any message when scanning the log from the indexed position. The latter could happen if the log is truncated
+ * after we get the indexed position but before we scan the log from there. In this case we simply return None and the
+ * caller will need to check on the truncated log and maybe retry or even do the search on another log segment.
+ *
+ * @param timestampMs The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(long timestampMs, long startingOffset) throws IOException {
+ // Get the index entry with a timestamp less than or equal to the target timestamp
+ TimestampOffset timestampOffset = timeIndex().lookup(timestampMs);
+ int position = offsetIndex().lookup(Math.max(timestampOffset.offset, startingOffset)).position;
+
+ // Search the timestamp
+ return Optional.ofNullable(log.searchForTimestamp(timestampMs, position, startingOffset));
+ }
+
+ /**
+ * Close this log segment
+ */
+ public void close() throws IOException {
+ if (maxTimestampAndOffsetSoFar != TimestampOffset.UNKNOWN)
+ Utils.swallow(LOGGER, Level.WARN, "maybeAppend", () -> timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true));
+ Utils.closeQuietly(lazyOffsetIndex, "offsetIndex", LOGGER);
+ Utils.closeQuietly(lazyTimeIndex, "timeIndex", LOGGER);
+ Utils.closeQuietly(log, "log", LOGGER);
+ Utils.closeQuietly(txnIndex, "txnIndex", LOGGER);
+ }
+
+ /**
+ * Close file handlers used by the log segment but don't write to disk. This is used when the disk may have failed
+ */
+ public void closeHandlers() {
+ Utils.swallow(LOGGER, Level.WARN, "offsetIndex", () -> lazyOffsetIndex.closeHandler());
+ Utils.swallow(LOGGER, Level.WARN, "timeIndex", () -> lazyTimeIndex.closeHandler());
+ Utils.swallow(LOGGER, Level.WARN, "log", () -> log.closeHandlers());
+ Utils.closeQuietly(txnIndex, "txnIndex", LOGGER);
+ }
+
+ /**
+ * Delete this log segment from the filesystem.
+ */
+ public void deleteIfExists() throws IOException {
+ try {
+ Utils.tryAll(asList(
+ () -> deleteTypeIfExists(() -> log.deleteIfExists(), "log", log.file(), true),
+ () -> deleteTypeIfExists(() -> lazyOffsetIndex.deleteIfExists(), "offset index", offsetIndexFile(), true),
+ () -> deleteTypeIfExists(() -> lazyTimeIndex.deleteIfExists(), "time index", timeIndexFile(), true),
+ () -> deleteTypeIfExists(() -> txnIndex.deleteIfExists(), "transaction index", txnIndex.file(), false)));
+ } catch (Throwable t) {
+ if (t instanceof IOException)
+ throw (IOException) t;
+ if (t instanceof Error)
+ throw (Error) t;
+ if (t instanceof RuntimeException)
+ throw (RuntimeException) t;
+ throw new IllegalStateException("Unexpected exception: " + t.getMessage(), t);
+ }
+ }
+
+ // Helper method for `deleteIfExists()`
+ private Void deleteTypeIfExists(StorageAction<Boolean, IOException> delete, String fileType, File file, boolean logIfMissing) throws IOException {
+ try {
+ if (delete.execute())
+ LOGGER.info("Deleted {} {}.", fileType, file.getAbsolutePath());
+ else if (logIfMissing)
+ LOGGER.info("Failed to delete {} {} because it does not exist.", fileType, file.getAbsolutePath());
+ return null;
+ } catch (IOException e) {
+ throw new IOException("Delete of " + fileType + " " + file.getAbsolutePath() + " failed.", e);
+ }
+ }
+
+ public boolean deleted() {
+ return !log.file().exists() && !offsetIndexFile().exists() && !timeIndexFile().exists() && !txnIndex.file().exists();
+ }
+
+
+ /**
+ * The last modified time of this log segment as a unix time stamp
+ */
+ public long lastModified() {
+ return log.file().lastModified();
+ }
+
+ /**
+ * The largest timestamp this segment contains, if maxTimestampSoFar >= 0, otherwise None.
+ */
+ public OptionalLong largestRecordTimestamp() {
+ long maxTimestampSoFar = maxTimestampSoFar();
+ if (maxTimestampSoFar >= 0)
+ return OptionalLong.of(maxTimestampSoFar);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * The largest timestamp this segment contains.
+ */
+ public long largestTimestamp() {
+ long maxTimestampSoFar = maxTimestampSoFar();
+ if (maxTimestampSoFar >= 0)
+ return maxTimestampSoFar;
+ return lastModified();
+ }
+
+ /**
+ * Change the last modified time for this log segment
+ */
+ public void setLastModified(long ms) throws IOException {
+ FileTime fileTime = FileTime.fromMillis(ms);
+ Files.setLastModifiedTime(log.file().toPath(), fileTime);
+ Files.setLastModifiedTime(offsetIndexFile().toPath(), fileTime);
+ Files.setLastModifiedTime(timeIndexFile().toPath(), fileTime);
+ }
+
+ public static LogSegment open(File dir, long baseOffset, LogConfig config, Time time, int initFileSize, boolean preallocate) throws IOException {
+ return open(dir, baseOffset, config, time, false, initFileSize, preallocate, "");
+ }
+
+ public static LogSegment open(File dir, long baseOffset, LogConfig config, Time time, boolean fileAlreadyExists,
+ int initFileSize, boolean preallocate, String fileSuffix) throws IOException {
+ int maxIndexSize = config.maxIndexSize;
+ return new LogSegment(
+ FileRecords.open(LogFileUtils.logFile(dir, baseOffset, fileSuffix), fileAlreadyExists, initFileSize, preallocate),
+ LazyIndex.forOffset(LogFileUtils.offsetIndexFile(dir, baseOffset, fileSuffix), baseOffset, maxIndexSize),
+ LazyIndex.forTime(LogFileUtils.timeIndexFile(dir, baseOffset, fileSuffix), baseOffset, maxIndexSize),
+ new TransactionIndex(baseOffset, LogFileUtils.transactionIndexFile(dir, baseOffset, fileSuffix)),
+ baseOffset,
+ config.indexInterval,
+ config.randomSegmentJitter(),
+ time);
+ }
+
+ public static void deleteIfExists(File dir, long baseOffset, String fileSuffix) throws IOException {
Review Comment:
I noticed that you didn't implement the case where fileSuffix is not provided because we are always providing a suffix. Nice!
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
Do we need this as an interface level method? Asking because interface already exposes batchIterator(). Calculating lastBatch() is a helper function and we could just have it as a concrete implementation in AbstractRecords. Note that existing method firstBatch() is also not present in the interface.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + largestRecordTimestamp() +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
+ }
+ }
+
+ /**
+ * Update the directory reference for the log and indices in this segment. This would typically be called after a
+ * directory is renamed.
+ */
+ public void updateParentDir(File dir) {
+ log.updateParentDir(dir);
+ lazyOffsetIndex.updateParentDir(dir);
+ lazyTimeIndex.updateParentDir(dir);
+ txnIndex.updateParentDir(dir);
+ }
+
+ /**
+ * Change the suffix for the index and log files for this log segment
+ * IOException from this method should be handled by the caller
+ */
+ public void changeFileSuffixes(String oldSuffix, String newSuffix) throws IOException {
+ log.renameTo(new File(Utils.replaceSuffix(log.file().getPath(), oldSuffix, newSuffix)));
+ lazyOffsetIndex.renameTo(new File(Utils.replaceSuffix(offsetIndexFile().getPath(), oldSuffix, newSuffix)));
+ lazyTimeIndex.renameTo(new File(Utils.replaceSuffix(timeIndexFile().getPath(), oldSuffix, newSuffix)));
+ txnIndex.renameTo(new File(Utils.replaceSuffix(txnIndex.file().getPath(), oldSuffix, newSuffix)));
+ }
+
+ public boolean hasSuffix(String suffix) {
+ return log.file().getName().endsWith(suffix) &&
+ offsetIndexFile().getName().endsWith(suffix) &&
+ timeIndexFile().getName().endsWith(suffix) &&
+ txnIndex.file().getName().endsWith(suffix);
+ }
+
+ /**
+ * Append the largest time index entry to the time index and trim the log and indexes.
+ *
+ * The time index entry appended will be used to decide when to delete the segment.
+ */
+ public void onBecomeInactiveSegment() throws IOException {
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ offsetIndex().trimToValidSize();
+ timeIndex().trimToValidSize();
+ log.trim();
+ }
+
+ /**
+ * If not previously loaded,
+ * load the timestamp of the first message into memory.
+ */
+ private void loadFirstBatchTimestamp() {
+ if (!rollingBasedTimestamp.isPresent()) {
+ Iterator<FileChannelRecordBatch> iter = log.batches().iterator();
+ if (iter.hasNext())
+ rollingBasedTimestamp = OptionalLong.of(iter.next().maxTimestamp());
+ }
+ }
+
+ /**
+ * The time this segment has waited to be rolled.
+ * If the first message batch has a timestamp we use its timestamp to determine when to roll a segment. A segment
+ * is rolled if the difference between the new batch's timestamp and the first batch's timestamp exceeds the
+ * segment rolling time.
+ * If the first batch does not have a timestamp, we use the wall clock time to determine when to roll a segment. A
+ * segment is rolled if the difference between the current wall clock time and the segment create time exceeds the
+ * segment rolling time.
+ */
+ public long timeWaitedForRoll(long now, long messageTimestamp) {
+ // Load the timestamp of the first message into memory
+ loadFirstBatchTimestamp();
+ long ts = rollingBasedTimestamp.orElse(-1L);
+ if (ts >= 0)
+ return messageTimestamp - ts;
+ return now - created;
+ }
+
+ /**
+ * @return the first batch timestamp if the timestamp is available. Otherwise return Long.MaxValue
+ */
+ public long getFirstBatchTimestamp() {
+ loadFirstBatchTimestamp();
+ OptionalLong timestamp = rollingBasedTimestamp;
+ if (timestamp.isPresent() && timestamp.getAsLong() >= 0)
+ return timestamp.getAsLong();
+ return Long.MAX_VALUE;
+ }
+
+ /**
+ * Search the message offset based on timestamp and offset.
+ *
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ *
+ * - If all the messages in the segment have smaller offsets, return None
+ * - If all the messages in the segment have smaller timestamps, return None
+ * - If all the messages in the segment have larger timestamps, or no message in the segment has a timestamp
+ * the returned the offset will be max(the base offset of the segment, startingOffset) and the timestamp will be Message.NoTimestamp.
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * This method only returns None when 1) all messages' offset < startOffing or 2) the log is not empty but we did not
+ * see any message when scanning the log from the indexed position. The latter could happen if the log is truncated
+ * after we get the indexed position but before we scan the log from there. In this case we simply return None and the
+ * caller will need to check on the truncated log and maybe retry or even do the search on another log segment.
+ *
+ * @param timestampMs The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(long timestampMs, long startingOffset) throws IOException {
+ // Get the index entry with a timestamp less than or equal to the target timestamp
+ TimestampOffset timestampOffset = timeIndex().lookup(timestampMs);
+ int position = offsetIndex().lookup(Math.max(timestampOffset.offset, startingOffset)).position;
+
+ // Search the timestamp
+ return Optional.ofNullable(log.searchForTimestamp(timestampMs, position, startingOffset));
+ }
+
+ /**
+ * Close this log segment
+ */
+ public void close() throws IOException {
+ if (maxTimestampAndOffsetSoFar != TimestampOffset.UNKNOWN)
+ Utils.swallow(LOGGER, Level.WARN, "maybeAppend", () -> timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true));
+ Utils.closeQuietly(lazyOffsetIndex, "offsetIndex", LOGGER);
+ Utils.closeQuietly(lazyTimeIndex, "timeIndex", LOGGER);
+ Utils.closeQuietly(log, "log", LOGGER);
+ Utils.closeQuietly(txnIndex, "txnIndex", LOGGER);
+ }
+
+ /**
+ * Close file handlers used by the log segment but don't write to disk. This is used when the disk may have failed
+ */
+ public void closeHandlers() {
+ Utils.swallow(LOGGER, Level.WARN, "offsetIndex", () -> lazyOffsetIndex.closeHandler());
+ Utils.swallow(LOGGER, Level.WARN, "timeIndex", () -> lazyTimeIndex.closeHandler());
+ Utils.swallow(LOGGER, Level.WARN, "log", () -> log.closeHandlers());
+ Utils.closeQuietly(txnIndex, "txnIndex", LOGGER);
+ }
+
+ /**
+ * Delete this log segment from the filesystem.
+ */
+ public void deleteIfExists() throws IOException {
+ try {
+ Utils.tryAll(asList(
+ () -> deleteTypeIfExists(() -> log.deleteIfExists(), "log", log.file(), true),
+ () -> deleteTypeIfExists(() -> lazyOffsetIndex.deleteIfExists(), "offset index", offsetIndexFile(), true),
+ () -> deleteTypeIfExists(() -> lazyTimeIndex.deleteIfExists(), "time index", timeIndexFile(), true),
+ () -> deleteTypeIfExists(() -> txnIndex.deleteIfExists(), "transaction index", txnIndex.file(), false)));
+ } catch (Throwable t) {
+ if (t instanceof IOException)
+ throw (IOException) t;
+ if (t instanceof Error)
+ throw (Error) t;
+ if (t instanceof RuntimeException)
+ throw (RuntimeException) t;
+ throw new IllegalStateException("Unexpected exception: " + t.getMessage(), t);
+ }
+ }
+
+ // Helper method for `deleteIfExists()`
+ private Void deleteTypeIfExists(StorageAction<Boolean, IOException> delete, String fileType, File file, boolean logIfMissing) throws IOException {
+ try {
+ if (delete.execute())
+ LOGGER.info("Deleted {} {}.", fileType, file.getAbsolutePath());
+ else if (logIfMissing)
+ LOGGER.info("Failed to delete {} {} because it does not exist.", fileType, file.getAbsolutePath());
+ return null;
+ } catch (IOException e) {
+ throw new IOException("Delete of " + fileType + " " + file.getAbsolutePath() + " failed.", e);
+ }
+ }
+
+ public boolean deleted() {
Review Comment:
Is this only visible for testing? If yes, please add an annotation.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
Review Comment:
I understand that this refers to compatibility of metrics namespace compatibility which is changes on moving this file. Please make this more explicit here as it's not obvious what compatibility are we talking about here.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
Review Comment:
implements `AutoCloseable` ? This class holds reference to Mmapp'ed indexes and should be closed to release system resources.
(if you add this, please remember to add `@Overrride` to close method)
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
Review Comment:
Do we need a deep clone() method here instead of having a constructor that generates an object from another instance?
##########
clients/src/main/java/org/apache/kafka/common/record/AbstractRecords.java:
##########
@@ -44,6 +45,17 @@ public RecordBatch firstBatch() {
return iterator.next();
}
+ @Override
+ public Optional<RecordBatch> lastBatch() {
+ Iterator<? extends RecordBatch> iterator = batches().iterator();
+
+ RecordBatch batch = null;
+ while (iterator.hasNext())
+ batch = iterator.next();
+
+ return Optional.ofNullable(batch);
Review Comment:
For my own knowledge, why aren't we using Optional.empty() when value doesn't exist and choosing to have a nullable optional? Is that something to do with scala Option compatibility?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
Review Comment:
Can this be moved to default access since only UnifiedLog uses it? I would also request you to check other methods and if only the tests or UnifiedLog or LogSegments requires it, move it to default. It helps ensure that we don't have implementation leaks outside the log layer.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
Review Comment:
could be default access
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + largestRecordTimestamp() +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
Review Comment:
1. This is a new change, right? Can you please explain how IllegalState is handled?
2. Do we need to print the entire stack trace in the exception message? Perhaps simply "Unexpected exception thrown" should suffice (since we are already passing the entire exception)
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.timestamp;
+ }
+
+ public long offsetOfMaxTimestampSoFar() {
+ return maxTimestampAndOffsetSoFar.offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ public boolean canConvertToRelativeOffset(long offset) throws IOException {
Review Comment:
this should be private. Only this class uses it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1357210089
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java:
##########
@@ -41,7 +41,7 @@
* Methods of this class are thread safe. Make sure to check `AbstractIndex` subclasses
* documentation to establish their thread safety.
*/
-public class LazyIndex<T extends AbstractIndex> {
+public class LazyIndex<T extends AbstractIndex> implements Closeable {
Review Comment:
Ah, `Closeable extends AutoCloseable` and not the other way round. My bad!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358592259
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
Review Comment:
The indexes are memory mapped, so in theory it should be consistent for reads that go via the page cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358302231
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
`AbstractRecords` only exists because default methods weren't a thing then. Abstract classes that don't add any value like this one are a bit of anti-pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356894691
##########
clients/src/main/java/org/apache/kafka/common/record/Records.java:
##########
@@ -70,6 +71,13 @@ public interface Records extends TransferableRecords {
*/
AbstractIterator<? extends RecordBatch> batchIterator();
+ /**
+ * Return the last record batch if non-empty or an empty `Optional` otherwise.
+ *
+ * Note that this requires iterating over all the record batches and hence it's expensive.
+ */
+ Optional<RecordBatch> lastBatch();
Review Comment:
I don't see how it helps to only have it in `AbstractRecords`, the point is to encapsulate the implementation since the equivalent Java code is a lot more verbose than the Scala version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356974305
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java:
##########
@@ -41,7 +41,7 @@
* Methods of this class are thread safe. Make sure to check `AbstractIndex` subclasses
* documentation to establish their thread safety.
*/
-public class LazyIndex<T extends AbstractIndex> {
+public class LazyIndex<T extends AbstractIndex> implements Closeable {
Review Comment:
"Any object that implements java.lang.AutoCloseable, which includes all objects which implement java.io.Closeable, can be used as a resource."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1759795074
I believe the tests should pass this time, let's see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1355274637
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1945,26 +1947,17 @@ object UnifiedLog extends Logging {
logOffsetsListener)
}
- def logFile(dir: File, offset: Long, suffix: String = ""): File = LogFileUtils.logFile(dir, offset, suffix)
Review Comment:
This and other deleted methods were simply passing through to `LogFileUtils` and hence did not add enough value to retain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356884927
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
Review Comment:
Copy constructors are often preferable to `clone` methods and are used widely within the collections library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1356881661
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java:
##########
@@ -41,7 +41,7 @@
* Methods of this class are thread safe. Make sure to check `AbstractIndex` subclasses
* documentation to establish their thread safety.
*/
-public class LazyIndex<T extends AbstractIndex> {
+public class LazyIndex<T extends AbstractIndex> implements Closeable {
Review Comment:
Why is `AutoCloseable` better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358013464
##########
core/src/main/scala/kafka/utils/CoreUtils.scala:
##########
@@ -66,11 +66,11 @@ object CoreUtils {
action
} catch {
case e: Throwable => logLevel match {
- case Level.ERROR => logger.error(e.getMessage, e)
- case Level.WARN => logger.warn(e.getMessage, e)
- case Level.INFO => logger.info(e.getMessage, e)
- case Level.DEBUG => logger.debug(e.getMessage, e)
- case Level.TRACE => logger.trace(e.getMessage, e)
+ case Level.ERROR => logging.error(e.getMessage, e)
+ case Level.WARN => logging.warn(e.getMessage, e)
+ case Level.INFO => logging.info(e.getMessage, e)
+ case Level.DEBUG => logging.debug(e.getMessage, e)
+ case Level.TRACE => logging.trace(e.getMessage, e)
Review Comment:
do we want to add a test (perhaps for one of the functions that are using this utility) which could have caught this? Could be done as a separate JIRA.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
Review Comment:
I am assuming that not removing this timer from metric registry is ok because it is initialized once per JVM. Once JVM dies, the singleton registry dies along with it and thus, it is ok even if we don't remove it. Is that right?
Do you think it's worth adding a comment about this here? Asking because we explicitly try to remove metrics from registry in other classes where lifecycle of metricGroup is shorter than lifecycle of Kafka process.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /**
+ * The maximum timestamp we see so far.
+ *
+ * Note that this may result in time index materialization.
+ */
+ public long maxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().timestamp;
+ }
+
+ /**
+ * Note that this may result in time index materialization.
+ */
+ private long offsetOfMaxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ private boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ // We don't call `largestRecordTimestamp` below to avoid materializing the time index when `toString` is invoked
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + maxTimestampAndOffsetSoFar.timestamp +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
+ }
+ }
+
+ /**
+ * Update the directory reference for the log and indices in this segment. This would typically be called after a
+ * directory is renamed.
+ */
+ public void updateParentDir(File dir) {
Review Comment:
could be default access. only used by log segments.
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /**
+ * The maximum timestamp we see so far.
+ *
+ * Note that this may result in time index materialization.
+ */
+ public long maxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().timestamp;
+ }
+
+ /**
+ * Note that this may result in time index materialization.
+ */
+ private long offsetOfMaxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ private boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ // We don't call `largestRecordTimestamp` below to avoid materializing the time index when `toString` is invoked
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + maxTimestampAndOffsetSoFar.timestamp +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
+ }
+ }
+
+ /**
+ * Update the directory reference for the log and indices in this segment. This would typically be called after a
+ * directory is renamed.
+ */
+ public void updateParentDir(File dir) {
+ log.updateParentDir(dir);
+ lazyOffsetIndex.updateParentDir(dir);
+ lazyTimeIndex.updateParentDir(dir);
+ txnIndex.updateParentDir(dir);
+ }
+
+ /**
+ * Change the suffix for the index and log files for this log segment
+ * IOException from this method should be handled by the caller
+ */
+ public void changeFileSuffixes(String oldSuffix, String newSuffix) throws IOException {
+ log.renameTo(new File(Utils.replaceSuffix(log.file().getPath(), oldSuffix, newSuffix)));
+ lazyOffsetIndex.renameTo(new File(Utils.replaceSuffix(offsetIndexFile().getPath(), oldSuffix, newSuffix)));
+ lazyTimeIndex.renameTo(new File(Utils.replaceSuffix(timeIndexFile().getPath(), oldSuffix, newSuffix)));
+ txnIndex.renameTo(new File(Utils.replaceSuffix(txnIndex.file().getPath(), oldSuffix, newSuffix)));
+ }
+
+ public boolean hasSuffix(String suffix) {
+ return log.file().getName().endsWith(suffix) &&
+ offsetIndexFile().getName().endsWith(suffix) &&
+ timeIndexFile().getName().endsWith(suffix) &&
+ txnIndex.file().getName().endsWith(suffix);
+ }
+
+ /**
+ * Append the largest time index entry to the time index and trim the log and indexes.
+ *
+ * The time index entry appended will be used to decide when to delete the segment.
+ */
+ public void onBecomeInactiveSegment() throws IOException {
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ offsetIndex().trimToValidSize();
+ timeIndex().trimToValidSize();
+ log.trim();
+ }
+
+ /**
+ * If not previously loaded,
+ * load the timestamp of the first message into memory.
+ */
+ private void loadFirstBatchTimestamp() {
+ if (!rollingBasedTimestamp.isPresent()) {
+ Iterator<FileChannelRecordBatch> iter = log.batches().iterator();
+ if (iter.hasNext())
+ rollingBasedTimestamp = OptionalLong.of(iter.next().maxTimestamp());
+ }
+ }
+
+ /**
+ * The time this segment has waited to be rolled.
+ * If the first message batch has a timestamp we use its timestamp to determine when to roll a segment. A segment
+ * is rolled if the difference between the new batch's timestamp and the first batch's timestamp exceeds the
+ * segment rolling time.
+ * If the first batch does not have a timestamp, we use the wall clock time to determine when to roll a segment. A
+ * segment is rolled if the difference between the current wall clock time and the segment create time exceeds the
+ * segment rolling time.
+ */
+ public long timeWaitedForRoll(long now, long messageTimestamp) {
+ // Load the timestamp of the first message into memory
+ loadFirstBatchTimestamp();
+ long ts = rollingBasedTimestamp.orElse(-1L);
+ if (ts >= 0)
+ return messageTimestamp - ts;
+ return now - created;
+ }
+
+ /**
+ * @return the first batch timestamp if the timestamp is available. Otherwise return Long.MaxValue
+ */
+ public long getFirstBatchTimestamp() {
+ loadFirstBatchTimestamp();
+ OptionalLong timestamp = rollingBasedTimestamp;
+ if (timestamp.isPresent() && timestamp.getAsLong() >= 0)
+ return timestamp.getAsLong();
+ return Long.MAX_VALUE;
+ }
+
+ /**
+ * Search the message offset based on timestamp and offset.
+ *
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ *
+ * - If all the messages in the segment have smaller offsets, return None
+ * - If all the messages in the segment have smaller timestamps, return None
+ * - If all the messages in the segment have larger timestamps, or no message in the segment has a timestamp
+ * the returned the offset will be max(the base offset of the segment, startingOffset) and the timestamp will be Message.NoTimestamp.
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * This method only returns None when 1) all messages' offset < startOffing or 2) the log is not empty but we did not
+ * see any message when scanning the log from the indexed position. The latter could happen if the log is truncated
+ * after we get the indexed position but before we scan the log from there. In this case we simply return None and the
+ * caller will need to check on the truncated log and maybe retry or even do the search on another log segment.
+ *
+ * @param timestampMs The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(long timestampMs, long startingOffset) throws IOException {
+ // Get the index entry with a timestamp less than or equal to the target timestamp
+ TimestampOffset timestampOffset = timeIndex().lookup(timestampMs);
+ int position = offsetIndex().lookup(Math.max(timestampOffset.offset, startingOffset)).position;
+
+ // Search the timestamp
+ return Optional.ofNullable(log.searchForTimestamp(timestampMs, position, startingOffset));
+ }
+
+ /**
+ * Close this log segment
+ */
+ public void close() throws IOException {
+ if (maxTimestampAndOffsetSoFar != TimestampOffset.UNKNOWN)
+ Utils.swallow(LOGGER, Level.WARN, "maybeAppend", () -> timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true));
+ Utils.closeQuietly(lazyOffsetIndex, "offsetIndex", LOGGER);
+ Utils.closeQuietly(lazyTimeIndex, "timeIndex", LOGGER);
+ Utils.closeQuietly(log, "log", LOGGER);
+ Utils.closeQuietly(txnIndex, "txnIndex", LOGGER);
+ }
+
+ /**
+ * Close file handlers used by the log segment but don't write to disk. This is used when the disk may have failed
+ */
+ public void closeHandlers() {
Review Comment:
only used by `LogSegments`. could be default access.
##########
core/src/test/scala/unit/kafka/log/LogCleanerParameterizedIntegrationTest.scala:
##########
@@ -120,12 +120,13 @@ class LogCleanerParameterizedIntegrationTest extends AbstractLogCleanerIntegrati
// Set the last modified time to an old value to force deletion of old segments
val endOffset = log.logEndOffset
- log.logSegments.foreach(_.lastModified = time.milliseconds - (2 * retentionMs))
+ log.logSegments.forEach(_.setLastModified(time.milliseconds - (2 * retentionMs)))
TestUtils.waitUntilTrue(() => log.logStartOffset == endOffset,
"Timed out waiting for deletion of old segments")
assertEquals(1, log.numberOfSegments)
cleaner.shutdown()
+ closeLog(log)
Review Comment:
Good catch!
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,887 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility - this metrics group was previously defined within
+ // a Scala object named `kafka.log.LogFlushStats`
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ /**
+ * The first time this is invoked, it will result in a time index lookup (including potential materialization of
+ * the time index).
+ */
+ public TimestampOffset readMaxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /**
+ * The maximum timestamp we see so far.
+ *
+ * Note that this may result in time index materialization.
+ */
+ public long maxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().timestamp;
+ }
+
+ /**
+ * Note that this may result in time index materialization.
+ */
+ private long offsetOfMaxTimestampSoFar() throws IOException {
+ return readMaxTimestampAndOffsetSoFar().offset;
+ }
+
+ /* Return the size in bytes of this log segment */
+ public int size() {
+ return log.sizeInBytes();
+ }
+
+ /**
+ * checks that the argument offset can be represented as an integer offset relative to the baseOffset.
+ */
+ private boolean canConvertToRelativeOffset(long offset) throws IOException {
+ return offsetIndex().canAppendOffset(offset);
+ }
+
+ /**
+ * Append the given messages starting with the given offset. Add
+ * an entry to the index if needed.
+ *
+ * It is assumed this method is being called from within a lock, it is not thread-safe otherwise.
+ *
+ * @param largestOffset The last offset in the message set
+ * @param largestTimestampMs The largest timestamp in the message set.
+ * @param shallowOffsetOfMaxTimestamp The offset of the message that has the largest timestamp in the messages to append.
+ * @param records The log entries to append.
+ * @throws LogSegmentOffsetOverflowException if the largest offset causes index offset overflow
+ */
+ public void append(long largestOffset,
+ long largestTimestampMs,
+ long shallowOffsetOfMaxTimestamp,
+ MemoryRecords records) throws IOException {
+ if (records.sizeInBytes() > 0) {
+ LOGGER.trace("Inserting {} bytes at end offset {} at position {} with largest timestamp {} at shallow offset {}",
+ records.sizeInBytes(), largestOffset, log.sizeInBytes(), largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ int physicalPosition = log.sizeInBytes();
+ if (physicalPosition == 0)
+ rollingBasedTimestamp = OptionalLong.of(largestTimestampMs);
+
+ ensureOffsetInRange(largestOffset);
+
+ // append the messages
+ long appendedBytes = log.append(records);
+ LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes, log.file(), largestOffset);
+ // Update the in memory max timestamp and corresponding offset.
+ if (largestTimestampMs > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
+ }
+ // append an entry to the index (if needed)
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(largestOffset, physicalPosition);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ bytesSinceLastIndexEntry = 0;
+ }
+ bytesSinceLastIndexEntry += records.sizeInBytes();
+ }
+ }
+
+ private void ensureOffsetInRange(long offset) throws IOException {
+ if (!canConvertToRelativeOffset(offset))
+ throw new LogSegmentOffsetOverflowException(this, offset);
+ }
+
+ private int appendChunkFromFile(FileRecords records, int position, BufferSupplier bufferSupplier) throws IOException {
+ int bytesToAppend = 0;
+ long maxTimestamp = Long.MIN_VALUE;
+ long offsetOfMaxTimestamp = Long.MIN_VALUE;
+ long maxOffset = Long.MIN_VALUE;
+ ByteBuffer readBuffer = bufferSupplier.get(1024 * 1024);
+
+ // find all batches that are valid to be appended to the current log segment and
+ // determine the maximum offset and timestamp
+ Iterator<FileChannelRecordBatch> nextBatches = records.batchesFrom(position).iterator();
+ FileChannelRecordBatch batch;
+ while ((batch = nextAppendableBatch(nextBatches, readBuffer, bytesToAppend)) != null) {
+ if (batch.maxTimestamp() > maxTimestamp) {
+ maxTimestamp = batch.maxTimestamp();
+ offsetOfMaxTimestamp = batch.lastOffset();
+ }
+ maxOffset = batch.lastOffset();
+ bytesToAppend += batch.sizeInBytes();
+ }
+
+ if (bytesToAppend > 0) {
+ // Grow buffer if needed to ensure we copy at least one batch
+ if (readBuffer.capacity() < bytesToAppend)
+ readBuffer = bufferSupplier.get(bytesToAppend);
+
+ readBuffer.limit(bytesToAppend);
+ records.readInto(readBuffer, position);
+
+ append(maxOffset, maxTimestamp, offsetOfMaxTimestamp, MemoryRecords.readableRecords(readBuffer));
+ }
+
+ bufferSupplier.release(readBuffer);
+ return bytesToAppend;
+ }
+
+ private FileChannelRecordBatch nextAppendableBatch(Iterator<FileChannelRecordBatch> recordBatches,
+ ByteBuffer readBuffer,
+ int bytesToAppend) throws IOException {
+ if (recordBatches.hasNext()) {
+ FileChannelRecordBatch batch = recordBatches.next();
+ if (canConvertToRelativeOffset(batch.lastOffset()) &&
+ (bytesToAppend == 0 || bytesToAppend + batch.sizeInBytes() < readBuffer.capacity()))
+ return batch;
+ }
+ return null;
+ }
+
+ /**
+ * Append records from a file beginning at the given position until either the end of the file
+ * is reached or an offset is found which is too large to convert to a relative offset for the indexes.
+ *
+ * @return the number of bytes appended to the log (may be less than the size of the input if an
+ * offset is encountered which would overflow this segment)
+ */
+ public int appendFromFile(FileRecords records, int start) throws IOException {
+ int position = start;
+ BufferSupplier bufferSupplier = new BufferSupplier.GrowableBufferSupplier();
+ while (position < start + records.sizeInBytes()) {
+ int bytesAppended = appendChunkFromFile(records, position, bufferSupplier);
+ if (bytesAppended == 0)
+ return position - start;
+ position += bytesAppended;
+ }
+ return position - start;
+ }
+
+ /* not thread safe */
+ public void updateTxnIndex(CompletedTxn completedTxn, long lastStableOffset) throws IOException {
+ if (completedTxn.isAborted) {
+ LOGGER.trace("Writing aborted transaction {} to transaction index, last stable offset is {}", completedTxn, lastStableOffset);
+ txnIndex.append(new AbortedTxn(completedTxn, lastStableOffset));
+ }
+ }
+
+ private void updateProducerState(ProducerStateManager producerStateManager, RecordBatch batch) throws IOException {
+ if (batch.hasProducerId()) {
+ long producerId = batch.producerId();
+ ProducerAppendInfo appendInfo = producerStateManager.prepareUpdate(producerId, AppendOrigin.REPLICATION);
+ Optional<CompletedTxn> maybeCompletedTxn = appendInfo.append(batch, Optional.empty());
+ producerStateManager.update(appendInfo);
+ if (maybeCompletedTxn.isPresent()) {
+ CompletedTxn completedTxn = maybeCompletedTxn.get();
+ long lastStableOffset = producerStateManager.lastStableOffset(completedTxn);
+ updateTxnIndex(completedTxn, lastStableOffset);
+ producerStateManager.completeTxn(completedTxn);
+ }
+ }
+ producerStateManager.updateMapEndOffset(batch.lastOffset() + 1);
+ }
+
+ /**
+ * Equivalent to {@code translateOffset(offset, 0)}.
+ *
+ * See {@link #translateOffset(long, int)} for details.
+ */
+ public LogOffsetPosition translateOffset(long offset) throws IOException {
+ return translateOffset(offset, 0);
+ }
+
+ /**
+ * Find the physical file position for the first message with offset >= the requested offset.
+ *
+ * The startingFilePosition argument is an optimization that can be used if we already know a valid starting position
+ * in the file higher than the greatest-lower-bound from the index.
+ *
+ * This method is thread-safe.
+ *
+ * @param offset The offset we want to translate
+ * @param startingFilePosition A lower bound on the file position from which to begin the search. This is purely an optimization and
+ * when omitted, the search will begin at the position in the offset index.
+ * @return The position in the log storing the message with the least offset >= the requested offset and the size of the
+ * message or null if no message meets this criteria.
+ */
+ LogOffsetPosition translateOffset(long offset, int startingFilePosition) throws IOException {
+ OffsetPosition mapping = offsetIndex().lookup(offset);
+ return log.searchForOffsetWithSize(offset, Math.max(mapping.position, startingFilePosition));
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, size())}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize) throws IOException {
+ return read(startOffset, maxSize, size());
+ }
+
+ /**
+ * Equivalent to {@code read(startOffset, maxSize, maxPosition, false)}.
+ *
+ * See {@link #read(long, int, long, boolean)} for details.
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition) throws IOException {
+ return read(startOffset, maxSize, maxPosition, false);
+ }
+
+ /**
+ * Read a message set from this segment beginning with the first offset >= startOffset. The message set will include
+ * no more than maxSize bytes and will end before maxOffset if a maxOffset is specified.
+ *
+ * This method is thread-safe.
+ *
+ * @param startOffset A lower bound on the first offset to include in the message set we read
+ * @param maxSize The maximum number of bytes to include in the message set we read
+ * @param maxPosition The maximum position in the log segment that should be exposed for read
+ * @param minOneMessage If this is true, the first message will be returned even if it exceeds `maxSize` (if one exists)
+ *
+ * @return The fetched data and the offset metadata of the first message whose offset is >= startOffset,
+ * or null if the startOffset is larger than the largest offset in this log
+ */
+ public FetchDataInfo read(long startOffset, int maxSize, long maxPosition, boolean minOneMessage) throws IOException {
+ if (maxSize < 0)
+ throw new IllegalArgumentException("Invalid max size " + maxSize + " for log read from segment " + log);
+
+ LogOffsetPosition startOffsetAndSize = translateOffset(startOffset);
+
+ // if the start position is already off the end of the log, return null
+ if (startOffsetAndSize == null)
+ return null;
+
+ int startPosition = startOffsetAndSize.position;
+ LogOffsetMetadata offsetMetadata = new LogOffsetMetadata(startOffset, this.baseOffset, startPosition);
+
+ int adjustedMaxSize = maxSize;
+ if (minOneMessage)
+ adjustedMaxSize = Math.max(maxSize, startOffsetAndSize.size);
+
+ // return a log segment but with zero size in the case below
+ if (adjustedMaxSize == 0)
+ return new FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY);
+
+ // calculate the length of the message set to read based on whether or not they gave us a maxOffset
+ int fetchSize = Math.min((int) (maxPosition - startPosition), adjustedMaxSize);
+
+ return new FetchDataInfo(offsetMetadata, log.slice(startPosition, fetchSize),
+ adjustedMaxSize < startOffsetAndSize.size, Optional.empty());
+ }
+
+ public OptionalLong fetchUpperBoundOffset(OffsetPosition startOffsetPosition, int fetchSize) throws IOException {
+ Optional<OffsetPosition> position = offsetIndex().fetchUpperBoundOffset(startOffsetPosition, fetchSize);
+ if (position.isPresent())
+ return OptionalLong.of(position.get().offset);
+ return OptionalLong.empty();
+ }
+
+ /**
+ * Run recovery on the given segment. This will rebuild the index from the log file and lop off any invalid bytes
+ * from the end of the log and index.
+ *
+ * This method is not thread-safe.
+ *
+ * @param producerStateManager Producer state corresponding to the segment's base offset. This is needed to recover
+ * the transaction index.
+ * @param leaderEpochCache Optionally a cache for updating the leader epoch during recovery.
+ * @return The number of bytes truncated from the log
+ * @throws LogSegmentOffsetOverflowException if the log segment contains an offset that causes the index offset to overflow
+ */
+ public int recover(ProducerStateManager producerStateManager, Optional<LeaderEpochFileCache> leaderEpochCache) throws IOException {
+ offsetIndex().reset();
+ timeIndex().reset();
+ txnIndex.reset();
+ int validBytes = 0;
+ int lastIndexEntry = 0;
+ maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+ try {
+ for (RecordBatch batch : log.batches()) {
+ batch.ensureValid();
+ ensureOffsetInRange(batch.lastOffset());
+
+ // The max timestamp is exposed at the batch level, so no need to iterate the records
+ if (batch.maxTimestamp() > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new TimestampOffset(batch.maxTimestamp(), batch.lastOffset());
+ }
+
+ // Build offset index
+ if (validBytes - lastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), validBytes);
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar());
+ lastIndexEntry = validBytes;
+ }
+ validBytes += batch.sizeInBytes();
+
+ if (batch.magic() >= RecordBatch.MAGIC_VALUE_V2) {
+ leaderEpochCache.ifPresent(cache -> {
+ if (batch.partitionLeaderEpoch() >= 0 &&
+ (!cache.latestEpoch().isPresent() || batch.partitionLeaderEpoch() > cache.latestEpoch().getAsInt()))
+ cache.assign(batch.partitionLeaderEpoch(), batch.baseOffset());
+ });
+ updateProducerState(producerStateManager, batch);
+ }
+ }
+ } catch (CorruptRecordException | InvalidRecordException e) {
+ LOGGER.warn("Found invalid messages in log segment {} at byte offset {}: {}. {}", log.file().getAbsolutePath(),
+ validBytes, e.getMessage(), e.getCause());
+ }
+ int truncated = log.sizeInBytes() - validBytes;
+ if (truncated > 0)
+ LOGGER.debug("Truncated {} invalid bytes at the end of segment {} during recovery", truncated, log.file().getAbsolutePath());
+
+ log.truncateTo(validBytes);
+ offsetIndex().trimToValidSize();
+ // A normally closed segment always appends the biggest timestamp ever seen into log segment, we do this as well.
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ timeIndex().trimToValidSize();
+ return truncated;
+ }
+
+ /**
+ * Check whether the last offset of the last batch in this segment overflows the indexes.
+ */
+ public boolean hasOverflow() throws IOException {
+ long nextOffset = readNextOffset();
+ return nextOffset > baseOffset && !canConvertToRelativeOffset(nextOffset - 1);
+ }
+
+ public TxnIndexSearchResult collectAbortedTxns(long fetchOffset, long upperBoundOffset) {
+ return txnIndex.collectAbortedTxns(fetchOffset, upperBoundOffset);
+ }
+
+ @Override
+ public String toString() {
+ // We don't call `largestRecordTimestamp` below to avoid materializing the time index when `toString` is invoked
+ return "LogSegment(baseOffset=" + baseOffset +
+ ", size=" + size() +
+ ", lastModifiedTime=" + lastModified() +
+ ", largestRecordTimestamp=" + maxTimestampAndOffsetSoFar.timestamp +
+ ")";
+ }
+
+ /**
+ * Truncate off all index and log entries with offsets >= the given offset.
+ * If the given offset is larger than the largest message in this segment, do nothing.
+ *
+ * This method is not thread-safe.
+ *
+ * @param offset The offset to truncate to
+ * @return The number of log bytes truncated
+ */
+ public int truncateTo(long offset) throws IOException {
+ // Do offset translation before truncating the index to avoid needless scanning
+ // in case we truncate the full index
+ LogOffsetPosition mapping = translateOffset(offset);
+ OffsetIndex offsetIndex = offsetIndex();
+ TimeIndex timeIndex = timeIndex();
+
+ offsetIndex.truncateTo(offset);
+ timeIndex.truncateTo(offset);
+ txnIndex.truncateTo(offset);
+
+ // After truncation, reset and allocate more space for the (new currently active) index
+ offsetIndex.resize(offsetIndex.maxIndexSize());
+ timeIndex.resize(timeIndex.maxIndexSize());
+
+ int bytesTruncated;
+ if (mapping == null)
+ bytesTruncated = 0;
+ else
+ bytesTruncated = log.truncateTo(mapping.position);
+
+ if (log.sizeInBytes() == 0) {
+ created = time.milliseconds();
+ rollingBasedTimestamp = OptionalLong.empty();
+ }
+
+ bytesSinceLastIndexEntry = 0;
+ if (maxTimestampSoFar() >= 0)
+ maxTimestampAndOffsetSoFar = readLargestTimestamp();
+
+ return bytesTruncated;
+ }
+
+ private TimestampOffset readLargestTimestamp() throws IOException {
+ // Get the last time index entry. If the time index is empty, it will return (-1, baseOffset)
+ TimestampOffset lastTimeIndexEntry = timeIndex().lastEntry();
+ OffsetPosition offsetPosition = offsetIndex().lookup(lastTimeIndexEntry.offset);
+
+ // Scan the rest of the messages to see if there is a larger timestamp after the last time index entry.
+ FileRecords.TimestampAndOffset maxTimestampOffsetAfterLastEntry = log.largestTimestampAfter(offsetPosition.position);
+ if (maxTimestampOffsetAfterLastEntry.timestamp > lastTimeIndexEntry.timestamp)
+ return new TimestampOffset(maxTimestampOffsetAfterLastEntry.timestamp, maxTimestampOffsetAfterLastEntry.offset);
+
+ return lastTimeIndexEntry;
+ }
+
+ /**
+ * Calculate the offset that would be used for the next message to be append to this segment.
+ * Note that this is expensive.
+ *
+ * This method is thread-safe.
+ */
+ public long readNextOffset() throws IOException {
+ FetchDataInfo fetchData = read(offsetIndex().lastOffset(), log.sizeInBytes());
+ if (fetchData == null)
+ return baseOffset;
+ else
+ return fetchData.records.lastBatch()
+ .map(batch -> batch.nextOffset())
+ .orElse(baseOffset);
+ }
+
+ /**
+ * Flush this log segment to disk.
+ *
+ * This method is thread-safe.
+ */
+ public void flush() throws IOException {
+ try {
+ LOG_FLUSH_TIMER.time(new Callable<Void>() {
+ // lambdas cannot declare a more specific exception type, so we use an anonymous inner class
+ @Override
+ public Void call() throws IOException {
+ log.flush();
+ offsetIndex().flush();
+ timeIndex().flush();
+ txnIndex.flush();
+ return null;
+ }
+ });
+ } catch (Exception e) {
+ if (e instanceof IOException)
+ throw (IOException) e;
+ else if (e instanceof RuntimeException)
+ throw (RuntimeException) e;
+ else
+ throw new IllegalStateException("Unexpected exception thrown: " + e, e);
+ }
+ }
+
+ /**
+ * Update the directory reference for the log and indices in this segment. This would typically be called after a
+ * directory is renamed.
+ */
+ public void updateParentDir(File dir) {
+ log.updateParentDir(dir);
+ lazyOffsetIndex.updateParentDir(dir);
+ lazyTimeIndex.updateParentDir(dir);
+ txnIndex.updateParentDir(dir);
+ }
+
+ /**
+ * Change the suffix for the index and log files for this log segment
+ * IOException from this method should be handled by the caller
+ */
+ public void changeFileSuffixes(String oldSuffix, String newSuffix) throws IOException {
+ log.renameTo(new File(Utils.replaceSuffix(log.file().getPath(), oldSuffix, newSuffix)));
+ lazyOffsetIndex.renameTo(new File(Utils.replaceSuffix(offsetIndexFile().getPath(), oldSuffix, newSuffix)));
+ lazyTimeIndex.renameTo(new File(Utils.replaceSuffix(timeIndexFile().getPath(), oldSuffix, newSuffix)));
+ txnIndex.renameTo(new File(Utils.replaceSuffix(txnIndex.file().getPath(), oldSuffix, newSuffix)));
+ }
+
+ public boolean hasSuffix(String suffix) {
+ return log.file().getName().endsWith(suffix) &&
+ offsetIndexFile().getName().endsWith(suffix) &&
+ timeIndexFile().getName().endsWith(suffix) &&
+ txnIndex.file().getName().endsWith(suffix);
+ }
+
+ /**
+ * Append the largest time index entry to the time index and trim the log and indexes.
+ *
+ * The time index entry appended will be used to decide when to delete the segment.
+ */
+ public void onBecomeInactiveSegment() throws IOException {
+ timeIndex().maybeAppend(maxTimestampSoFar(), offsetOfMaxTimestampSoFar(), true);
+ offsetIndex().trimToValidSize();
+ timeIndex().trimToValidSize();
+ log.trim();
+ }
+
+ /**
+ * If not previously loaded,
+ * load the timestamp of the first message into memory.
+ */
+ private void loadFirstBatchTimestamp() {
+ if (!rollingBasedTimestamp.isPresent()) {
+ Iterator<FileChannelRecordBatch> iter = log.batches().iterator();
+ if (iter.hasNext())
+ rollingBasedTimestamp = OptionalLong.of(iter.next().maxTimestamp());
+ }
+ }
+
+ /**
+ * The time this segment has waited to be rolled.
+ * If the first message batch has a timestamp we use its timestamp to determine when to roll a segment. A segment
+ * is rolled if the difference between the new batch's timestamp and the first batch's timestamp exceeds the
+ * segment rolling time.
+ * If the first batch does not have a timestamp, we use the wall clock time to determine when to roll a segment. A
+ * segment is rolled if the difference between the current wall clock time and the segment create time exceeds the
+ * segment rolling time.
+ */
+ public long timeWaitedForRoll(long now, long messageTimestamp) {
+ // Load the timestamp of the first message into memory
+ loadFirstBatchTimestamp();
+ long ts = rollingBasedTimestamp.orElse(-1L);
+ if (ts >= 0)
+ return messageTimestamp - ts;
+ return now - created;
+ }
+
+ /**
+ * @return the first batch timestamp if the timestamp is available. Otherwise return Long.MaxValue
+ */
+ public long getFirstBatchTimestamp() {
Review Comment:
could be default access since only used by LogSegments which is in same package.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #14529:
URL: https://github.com/apache/kafka/pull/14529#discussion_r1358301179
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -0,0 +1,873 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.storage.internals.log;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.NoSuchFileException;
+import java.nio.file.attribute.FileTime;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalLong;
+import java.util.concurrent.Callable;
+import java.util.concurrent.TimeUnit;
+
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.Timer;
+import org.apache.kafka.common.InvalidRecordException;
+import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.record.FileLogInputStream.FileChannelRecordBatch;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.FileRecords.LogOffsetPosition;
+import org.apache.kafka.common.record.MemoryRecords;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.utils.BufferSupplier;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.metrics.KafkaMetricsGroup;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.slf4j.event.Level;
+
+import static java.util.Arrays.asList;
+
+/**
+ * A segment of the log. Each segment has two components: a log and an index. The log is a FileRecords containing
+ * the actual messages. The index is an OffsetIndex that maps from logical offsets to physical file positions. Each
+ * segment has a base offset which is an offset <= the least offset of any message in this segment and > any offset in
+ * any previous segment.
+ *
+ * A segment with a base offset of [base_offset] would be stored in two files, a [base_offset].index and a [base_offset].log file.
+ *
+ * This class is not thread-safe.
+ */
+public class LogSegment {
+ private static final Logger LOGGER = LoggerFactory.getLogger(LogSegment.class);
+ private static final Timer LOG_FLUSH_TIMER;
+
+ static {
+ KafkaMetricsGroup logFlushStatsMetricsGroup = new KafkaMetricsGroup(LogSegment.class) {
+ @Override
+ public MetricName metricName(String name, Map<String, String> tags) {
+ // Override the group and type names for compatibility
+ return KafkaMetricsGroup.explicitMetricName("kafka.log", "LogFlushStats", name, tags);
+ }
+ };
+ LOG_FLUSH_TIMER = logFlushStatsMetricsGroup.newTimer("LogFlushRateAndTimeMs", TimeUnit.MILLISECONDS, TimeUnit.SECONDS);
+ }
+
+ private final FileRecords log;
+ private final LazyIndex<OffsetIndex> lazyOffsetIndex;
+ private final LazyIndex<TimeIndex> lazyTimeIndex;
+ private final TransactionIndex txnIndex;
+ private final long baseOffset;
+ private final int indexIntervalBytes;
+ private final long rollJitterMs;
+ private final Time time;
+
+ // The timestamp we used for time based log rolling and for ensuring max compaction delay
+ // volatile for LogCleaner to see the update
+ private volatile OptionalLong rollingBasedTimestamp = OptionalLong.empty();
+
+ /* The maximum timestamp and offset we see so far */
+ private volatile TimestampOffset maxTimestampAndOffsetSoFar = TimestampOffset.UNKNOWN;
+
+ private long created;
+
+ /* the number of bytes since we last added an entry in the offset index */
+ private int bytesSinceLastIndexEntry = 0;
+
+ /**
+ * Create a LogSegment with the provided parameters.
+ *
+ * @param log The file records containing log entries
+ * @param lazyOffsetIndex The offset index
+ * @param lazyTimeIndex The timestamp index
+ * @param txnIndex The transaction index
+ * @param baseOffset A lower bound on the offsets in this segment
+ * @param indexIntervalBytes The approximate number of bytes between entries in the index
+ * @param rollJitterMs The maximum random jitter subtracted from the scheduled segment roll time
+ * @param time The time instance
+ */
+ public LogSegment(FileRecords log,
+ LazyIndex<OffsetIndex> lazyOffsetIndex,
+ LazyIndex<TimeIndex> lazyTimeIndex,
+ TransactionIndex txnIndex,
+ long baseOffset,
+ int indexIntervalBytes,
+ long rollJitterMs,
+ Time time) {
+ this.log = log;
+ this.lazyOffsetIndex = lazyOffsetIndex;
+ this.lazyTimeIndex = lazyTimeIndex;
+ this.txnIndex = txnIndex;
+ this.baseOffset = baseOffset;
+ this.indexIntervalBytes = indexIntervalBytes;
+ this.rollJitterMs = rollJitterMs;
+ this.time = time;
+ this.created = time.milliseconds();
+ }
+
+ // Visible for testing
+ public LogSegment(LogSegment segment) {
+ this(segment.log, segment.lazyOffsetIndex, segment.lazyTimeIndex, segment.txnIndex, segment.baseOffset,
+ segment.indexIntervalBytes, segment.rollJitterMs, segment.time);
+ }
+
+ public OffsetIndex offsetIndex() throws IOException {
+ return lazyOffsetIndex.get();
+ }
+
+ public File offsetIndexFile() {
+ return lazyOffsetIndex.file();
+ }
+
+ public TimeIndex timeIndex() throws IOException {
+ return lazyTimeIndex.get();
+ }
+
+ public File timeIndexFile() {
+ return lazyTimeIndex.file();
+ }
+
+ public long baseOffset() {
+ return baseOffset;
+ }
+
+ public FileRecords log() {
+ return log;
+ }
+
+ public long rollJitterMs() {
+ return rollJitterMs;
+ }
+
+ public TransactionIndex txnIndex() {
+ return txnIndex;
+ }
+
+ public boolean shouldRoll(RollParams rollParams) throws IOException {
+ boolean reachedRollMs = timeWaitedForRoll(rollParams.now, rollParams.maxTimestampInMessages) > rollParams.maxSegmentMs - rollJitterMs;
+ int size = size();
+ return size > rollParams.maxSegmentBytes - rollParams.messagesSize ||
+ (size > 0 && reachedRollMs) ||
+ offsetIndex().isFull() || timeIndex().isFull() || !canConvertToRelativeOffset(rollParams.maxOffsetInMessages);
+ }
+
+ public void resizeIndexes(int size) throws IOException {
+ offsetIndex().resize(size);
+ timeIndex().resize(size);
+ }
+
+ public void sanityCheck(boolean timeIndexFileNewlyCreated) throws IOException {
+ if (offsetIndexFile().exists()) {
+ // Resize the time index file to 0 if it is newly created.
+ if (timeIndexFileNewlyCreated)
+ timeIndex().resize(0);
+ // Sanity checks for time index and offset index are skipped because
+ // we will recover the segments above the recovery point in recoverLog()
+ // in any case so sanity checking them here is redundant.
+ txnIndex.sanityCheck();
+ } else
+ throw new NoSuchFileException("Offset index file " + offsetIndexFile().getAbsolutePath() + " does not exist");
+ }
+
+ public TimestampOffset maxTimestampAndOffsetSoFar() throws IOException {
+ if (maxTimestampAndOffsetSoFar == TimestampOffset.UNKNOWN)
+ maxTimestampAndOffsetSoFar = timeIndex().lastEntry();
+ return maxTimestampAndOffsetSoFar;
+ }
+
+ /* The maximum timestamp we see so far */
+ public long maxTimestampSoFar() {
Review Comment:
I did. There were exactly 3 cases, the ones you pointed out. You don't need to sell me on the benefit, I filed KAFKA-14489 from the start of this effort. ;)
I originally intended to do the `LogSegments` conversion in a separate PR and the 3 examples couldn't be default package at first. Once I added the `LogSegments` conversion to this PR, it opened up that opportunity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14481: Move LogSegment/LogSegments to storage module [kafka]
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on PR #14529:
URL: https://github.com/apache/kafka/pull/14529#issuecomment-1762008506
> @ijuma Thanks for the PR. I will review it tomorrow.
Thanks @satishd. I won't merge until Tuesday to give you a chance to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org