You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "satishd (via GitHub)" <gi...@apache.org> on 2023/03/31 12:25:48 UTC
[GitHub] [kafka] satishd opened a new pull request, #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
satishd opened a new pull request, #13487:
URL: https://github.com/apache/kafka/pull/13487
**_This PR is NOT YET READY for review. It needs some cleaning of the code and a few more changes._**
It is raised to run jenkins jobs with different environments. I will update the description once it is ready for review
Added functionality to copy log segments, indexes to the target remote storage for each topic partition enabled with tiered storage. This involves creating scheduled tasks for all leader partition replicas to copy their log segments in sequence to tiered storage.t push -f origin
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] Hangleton commented on pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "Hangleton (via GitHub)" <gi...@apache.org>.
Hangleton commented on PR #13487:
URL: https://github.com/apache/kafka/pull/13487#issuecomment-1508916524
> *Since this PR blocks other following PRs, I'm going to merge it soon. We can have a follow-up PR to address any new comments if any. Thank you.*
Provided a few comments FYI Luke (@showuon).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jlprat commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "jlprat (via GitHub)" <gi...@apache.org>.
jlprat commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1211296146
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(RLMTask.class);
+ }
+
+ boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The copiedOffsetOption is OptionalLong.empty() initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition " + topicIdPartition + " can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void copyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed (high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ ListIterator<LogSegment> logSegmentsIter = sortedSegments.subList(0, activeSegIndex).listIterator();
+ while (logSegmentsIter.hasNext()) {
+ LogSegment segment = logSegmentsIter.next();
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ copyLogSegment(log, segment, getNextSegmentBaseOffset(activeSegBaseOffset, logSegmentsIter));
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private long getNextSegmentBaseOffset(long activeSegBaseOffset, ListIterator<LogSegment> logSegmentsIter) {
+ long nextSegmentBaseOffset;
+ if (logSegmentsIter.hasNext()) {
+ nextSegmentBaseOffset = logSegmentsIter.next().baseOffset();
+ logSegmentsIter.previous();
+ } else {
+ nextSegmentBaseOffset = activeSegBaseOffset;
+ }
+
+ return nextSegmentBaseOffset;
+ }
+
+ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegmentBaseOffset) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long endOffset = nextSegmentBaseOffset - 1;
+ File producerStateSnapshotFile = log.producerStateManager().fetchSnapshot(nextSegmentBaseOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextSegmentBaseOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerStateSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ copyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+ }
+ );
+ convertToLeaderOrFollower.accept(rlmTaskWithFuture.rlmTask);
+ }
+
+ static class RLMTaskWithFuture {
+
+ private final RLMTask rlmTask;
+ private final Future<?> future;
+
+ RLMTaskWithFuture(RLMTask rlmTask, Future<?> future) {
+ this.rlmTask = rlmTask;
+ this.future = future;
+ }
+
+ public void cancel() {
+ rlmTask.cancel();
+ try {
+ future.cancel(true);
+ } catch (Exception ex) {
+ LOGGER.error("Error occurred while canceling the task: {}", rlmTask, ex);
+ }
+ }
+
+ }
+
+ /**
+ * Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
+ */
+ public void close() {
+ synchronized (this) {
+ if (!closed) {
+ leaderOrFollowerTasks.values().forEach(RLMTaskWithFuture::cancel);
+ Utils.closeQuietly(remoteLogStorageManager, "RemoteLogStorageManager");
+ Utils.closeQuietly(remoteLogMetadataManager, "RemoteLogMetadataManager");
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
+ try {
+ rlmScheduledThreadPool.shutdown();
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ leaderOrFollowerTasks.clear();
+ closed = true;
+ }
+ }
+ }
+
+ static class RLMScheduledThreadPool {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RLMScheduledThreadPool.class);
+ private final int poolSize;
+ private final ScheduledThreadPoolExecutor scheduledThreadPool;
+
+ public RLMScheduledThreadPool(int poolSize) {
+ this.poolSize = poolSize;
+ scheduledThreadPool = createPool();
+ }
+
+ private ScheduledThreadPoolExecutor createPool() {
+ ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(poolSize);
+ threadPool.setRemoveOnCancelPolicy(true);
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
+ threadPool.setThreadFactory(new ThreadFactory() {
+ private final AtomicInteger sequence = new AtomicInteger();
+
+ public Thread newThread(Runnable r) {
+ return KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r);
+ }
+ });
+
+ return threadPool;
+ }
+
+ public ScheduledFuture scheduleWithFixedDelay(Runnable runnable, long initialDelay, long delay, TimeUnit timeUnit) {
Review Comment:
I know I'm late to the party, but I can see that `ScheduledFuture` is parametrized, but used "raw" in here.
I think this should be like this instead:
> public ScheduledFuture<?> scheduleWithFixedDelay(Runnable runnable, long initialDelay, long delay, TimeUnit timeUnit) {
I'll submit a fix for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on PR #13487:
URL: https://github.com/apache/kafka/pull/13487#issuecomment-1501357857
Thanks @junrao for your review comments. Some of those comments are addressed inline or with the updated commits, will address the remaining few comments soon.
> @satishd : Do you plan to add any new tests?
As I mentioned in the earlier PR description, that was a 'todo' to be done in this PR. They are added now with the latest commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1158212079
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
Review Comment:
Ah, you're right!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1160112094
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,97 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false)
+ threadPool.setThreadFactory(new ThreadFactory {
+ private val sequence = new AtomicInteger()
+
+ override def newThread(r: Runnable): Thread = {
+ KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r)
+ }
+ })
+
+ threadPool
+ }
+
+ def resizePool(size: Int): Unit = {
+ info(s"Resizing pool from ${scheduledThreadPool.getCorePoolSize} to $size")
+ scheduledThreadPool.setCorePoolSize(size)
+ }
+
+ def poolSize(): Int = scheduledThreadPool.getCorePoolSize
+
+ def getIdlePercent(): Double = {
+ 1 - scheduledThreadPool.getActiveCount().asInstanceOf[Double] / scheduledThreadPool.getCorePoolSize.asInstanceOf[Double]
+ }
+
+ def scheduleWithFixedDelay(runnable: Runnable, initialDelay: Long, delay: Long,
+ timeUnit: TimeUnit): ScheduledFuture[_] = {
+ info(s"Scheduling runnable $runnable with initial delay: $initialDelay, fixed delay: $delay")
+ scheduledThreadPool.scheduleWithFixedDelay(runnable, initialDelay, delay, timeUnit)
+ }
+
+ def shutdown(): Boolean = {
+ info("Shutting down scheduled thread pool")
+ scheduledThreadPool.shutdownNow()
+ //waits for 2 mins to terminate the current tasks
+ scheduledThreadPool.awaitTermination(2, TimeUnit.MINUTES)
+ }
+}
+
/**
* This class is responsible for
- * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances.
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
* - receives any leader and follower replica events and partition stop events and act on them
- * - also provides APIs to fetch indexes, metadata about remote log segments.
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
*
* @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
* @param brokerId id of the current broker.
* @param logDir directory of Kafka log segments.
+ * @param time Time instance.
Review Comment:
This was an existing code in Scala in my other branch. But it avoids the review again during Java migration later. I did the Java migration as you suggested and pushed it to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161677863
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,736 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
Review Comment:
I didn't see any place we invoke this method. I think `ReplicaManager#becomeLeaderOrFollower` should be responsible to invoke this method, right?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,736 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(RLMTask.class);
+ }
+
+ boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The copiedOffsetOption is OptionalLong.empty() initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition " + topicIdPartition + " can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void copyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed (high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ copyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void copyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerStateSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerStateSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ copyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+ }
+ );
+ convertToLeaderOrFollower.accept(rlmTaskWithFuture.rlmTask);
+ }
+
+ static class RLMTaskWithFuture {
+
+ private final RLMTask rlmTask;
+ private final Future<?> future;
+
+ RLMTaskWithFuture(RLMTask rlmTask, Future<?> future) {
+ this.rlmTask = rlmTask;
+ this.future = future;
+ }
+
+ public void cancel() {
+ rlmTask.cancel();
+ try {
+ future.cancel(true);
+ } catch (Exception ex) {
+ LOGGER.error("Error occurred while canceling the task: {}", rlmTask, ex);
+ }
+ }
+
+ }
+
+ static class InMemoryLeaderEpochCheckpoint implements LeaderEpochCheckpoint {
Review Comment:
This is not necessary, and use `org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint` instead.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,736 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
Review Comment:
This is not necessary if we don't specify which exception will be thrown and when.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161286262
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
Review Comment:
This is already handled while finding the `fromOffset` like below.
```
long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161286452
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
Review Comment:
`CollectionConverters` is only available in Scala-2.13. We used `JavaConverters` to support both 2.12 and 2.13.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on PR #13487:
URL: https://github.com/apache/kafka/pull/13487#issuecomment-1499734582
@satishd : Do you plan to add any new tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161286998
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
Review Comment:
Does not `copySegmentFinishedRlsm` represents what it is meant for instead of `finishedSegmentRlsm `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1160112094
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,97 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false)
+ threadPool.setThreadFactory(new ThreadFactory {
+ private val sequence = new AtomicInteger()
+
+ override def newThread(r: Runnable): Thread = {
+ KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r)
+ }
+ })
+
+ threadPool
+ }
+
+ def resizePool(size: Int): Unit = {
+ info(s"Resizing pool from ${scheduledThreadPool.getCorePoolSize} to $size")
+ scheduledThreadPool.setCorePoolSize(size)
+ }
+
+ def poolSize(): Int = scheduledThreadPool.getCorePoolSize
+
+ def getIdlePercent(): Double = {
+ 1 - scheduledThreadPool.getActiveCount().asInstanceOf[Double] / scheduledThreadPool.getCorePoolSize.asInstanceOf[Double]
+ }
+
+ def scheduleWithFixedDelay(runnable: Runnable, initialDelay: Long, delay: Long,
+ timeUnit: TimeUnit): ScheduledFuture[_] = {
+ info(s"Scheduling runnable $runnable with initial delay: $initialDelay, fixed delay: $delay")
+ scheduledThreadPool.scheduleWithFixedDelay(runnable, initialDelay, delay, timeUnit)
+ }
+
+ def shutdown(): Boolean = {
+ info("Shutting down scheduled thread pool")
+ scheduledThreadPool.shutdownNow()
+ //waits for 2 mins to terminate the current tasks
+ scheduledThreadPool.awaitTermination(2, TimeUnit.MINUTES)
+ }
+}
+
/**
* This class is responsible for
- * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances.
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
* - receives any leader and follower replica events and partition stop events and act on them
- * - also provides APIs to fetch indexes, metadata about remote log segments.
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
*
* @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
* @param brokerId id of the current broker.
* @param logDir directory of Kafka log segments.
+ * @param time Time instance.
Review Comment:
This was an existing code in Scala in my other branch with a few changes. But your suggestion is better as it avoids the review again during Java migration later.
I did the Java migration as you suggested and pushed it to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1160223702
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
Review Comment:
update => the updated ?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
Review Comment:
This method doesn't seem to throw Exception? Also, could this be private?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
Review Comment:
None => empty?
Also, there is no readOffset. Should copiedOffsetOption be readOffset?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
Review Comment:
Wrong class?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ private void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+
Review Comment:
unnecessary extra line.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
Review Comment:
There is no tpId.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
Review Comment:
`findHighestRemoteOffset` could return -1. Should we handle that?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
Review Comment:
JavaConverters is deprecated. Could we use CollectionConverters instead?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
Review Comment:
Could this be private?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
Review Comment:
Instead of passing in null `Path`, would it be better to pass in `Optional`?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
Review Comment:
This seems a bit complicated. Could we just set toOffset to the min of lso and activeSegBaseOffset?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
Review Comment:
remove space before : ?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
Review Comment:
Could this be private?
##########
server-common/src/main/java/org/apache/kafka/server/common/CheckpointFile.java:
##########
@@ -77,20 +77,8 @@ public void write(Collection<T> entries) throws IOException {
// write to temp file and then swap with the existing file
try (FileOutputStream fileOutputStream = new FileOutputStream(tempPath.toFile());
BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(fileOutputStream, StandardCharsets.UTF_8))) {
- // Write the version
- writer.write(Integer.toString(version));
- writer.newLine();
-
- // Write the entries count
- writer.write(Integer.toString(entries.size()));
- writer.newLine();
-
- // Write each entry on a new line.
- for (T entry : entries) {
- writer.write(formatter.toString(entry));
- writer.newLine();
- }
-
+ CheckpointWriteBuffer<T> checkpointWriteBuffer = new CheckpointWriteBuffer<T>(writer, version, formatter);
Review Comment:
new CheckpointWriteBuffer<T> => new CheckpointWriteBuffer<> ?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ private void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+
+ }
+ );
+ convertToLeaderOrFollower.accept(rlmTaskWithFuture.rlmTask);
+ }
+
+ private static class RLMTaskWithFuture {
+
+ private final RLMTask rlmTask;
+ private final Future<?> future;
+
+ RLMTaskWithFuture(RLMTask rlmTask, Future<?> future) {
+ this.rlmTask = rlmTask;
+ this.future = future;
+ }
+
+ public void cancel() {
+ rlmTask.cancel();
+ try {
+ future.cancel(true);
+ } catch (Exception ex) {
+ LOGGER.error("Error occurred while canceling the task: {}", rlmTask, ex);
+ }
+ }
+
+ }
+
+ private static class InMemoryLeaderEpochCheckpoint implements LeaderEpochCheckpoint {
+ private List<EpochEntry> epochs = Collections.emptyList();
+
+ public void write(Collection<EpochEntry> epochs) {
+ this.epochs = new ArrayList<>(epochs);
+ }
+
+ public List<EpochEntry> read() {
+ return Collections.unmodifiableList(epochs);
+ }
+
+ public ByteBuffer readAsByteBuffer() throws IOException {
+ ByteArrayOutputStream stream = new ByteArrayOutputStream();
+ try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stream, StandardCharsets.UTF_8));) {
+ CheckpointFile.CheckpointWriteBuffer<EpochEntry> writeBuffer = new CheckpointFile.CheckpointWriteBuffer<>(writer, 0, LeaderEpochCheckpointFile.FORMATTER);
+ writeBuffer.write(epochs);
+ writer.flush();
+ }
+
+ return ByteBuffer.wrap(stream.toByteArray());
+ }
+ }
+
+ /**
+ * Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
+ */
+ public void close() {
+ synchronized (this) {
+ if (!closed) {
+ leaderOrFollowerTasks.values().forEach(RLMTaskWithFuture::cancel);
+ Utils.closeQuietly(remoteLogStorageManager, "RemoteLogStorageManager");
+ Utils.closeQuietly(remoteLogMetadataManager, "RemoteLogMetadataManager");
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
+ try {
+ rlmScheduledThreadPool.shutdown();
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ leaderOrFollowerTasks.clear();
+ closed = true;
+ }
+ }
+ }
+
+ private static class RLMScheduledThreadPool {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RLMScheduledThreadPool.class);
+ private final int poolSize;
+ private final ScheduledThreadPoolExecutor scheduledThreadPool;
+
+ public RLMScheduledThreadPool(int poolSize) {
+ this.poolSize = poolSize;
+ scheduledThreadPool = createPool();
+ }
+
+ private ScheduledThreadPoolExecutor createPool() {
+ ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(poolSize);
+ threadPool.setRemoveOnCancelPolicy(true);
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
+ threadPool.setThreadFactory(new ThreadFactory() {
+ private final AtomicInteger sequence = new AtomicInteger();
+
+ public Thread newThread(Runnable r) {
+ return KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r);
+ }
+ });
+
+ return threadPool;
+ }
+
+ public void resizePool(int size) {
Review Comment:
This seems unused?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
Review Comment:
Should we include the segment base offset too?
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -520,6 +522,11 @@ class UnifiedLog(@volatile var logStartOffset: Long,
localLog.updateRecoveryPoint(offset)
}
}
+ def updateHighestOffsetInRemoteStorage(offset: Long): Unit = {
+ if (!remoteLogEnabled())
+ warn(s"Unable to update the highest offset with remote index as: $offset, the existing value: $highestOffsetInRemoteStorage since remote storage is not enabled.")
Review Comment:
Could we make it more readable like the following?
Unable to update the highest offset in remote storage with offset $offset since remote storage is not enabled. The existing highest offset is $highestOffsetInRemoteStorage.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1231,10 +1238,10 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
remoteLogManager.get.findOffsetByTimestamp(topicPartition, targetTimestamp, logStartOffset, leaderEpochCache.get)
- } else None
+ } else Optional.empty()
Review Comment:
We should return None, right?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
Review Comment:
space before (
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
Review Comment:
`readNextOffset() ` could be a bit expensive. Could we get nextOffset from the base offset of the next segment?
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerStateManager.java:
##########
@@ -605,6 +609,8 @@ public Optional<SnapshotFile> removeAndMarkSnapshotForDeletion(long snapshotOffs
return Optional.empty();
}
+
Review Comment:
unnecessary new lines.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
Review Comment:
copySegmentFinishedRlsm => finishedSegmentRlsm ?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
Review Comment:
producerIdSnapshotFile => producerStateSnapshotFile ?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ private void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+
+ }
+ );
+ convertToLeaderOrFollower.accept(rlmTaskWithFuture.rlmTask);
+ }
+
+ private static class RLMTaskWithFuture {
+
+ private final RLMTask rlmTask;
+ private final Future<?> future;
+
+ RLMTaskWithFuture(RLMTask rlmTask, Future<?> future) {
+ this.rlmTask = rlmTask;
+ this.future = future;
+ }
+
+ public void cancel() {
+ rlmTask.cancel();
+ try {
+ future.cancel(true);
+ } catch (Exception ex) {
+ LOGGER.error("Error occurred while canceling the task: {}", rlmTask, ex);
+ }
+ }
+
+ }
+
+ private static class InMemoryLeaderEpochCheckpoint implements LeaderEpochCheckpoint {
+ private List<EpochEntry> epochs = Collections.emptyList();
+
+ public void write(Collection<EpochEntry> epochs) {
+ this.epochs = new ArrayList<>(epochs);
+ }
+
+ public List<EpochEntry> read() {
+ return Collections.unmodifiableList(epochs);
+ }
+
+ public ByteBuffer readAsByteBuffer() throws IOException {
+ ByteArrayOutputStream stream = new ByteArrayOutputStream();
+ try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stream, StandardCharsets.UTF_8));) {
+ CheckpointFile.CheckpointWriteBuffer<EpochEntry> writeBuffer = new CheckpointFile.CheckpointWriteBuffer<>(writer, 0, LeaderEpochCheckpointFile.FORMATTER);
+ writeBuffer.write(epochs);
+ writer.flush();
+ }
+
+ return ByteBuffer.wrap(stream.toByteArray());
+ }
+ }
+
+ /**
+ * Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
+ */
+ public void close() {
+ synchronized (this) {
+ if (!closed) {
+ leaderOrFollowerTasks.values().forEach(RLMTaskWithFuture::cancel);
+ Utils.closeQuietly(remoteLogStorageManager, "RemoteLogStorageManager");
+ Utils.closeQuietly(remoteLogMetadataManager, "RemoteLogMetadataManager");
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
+ try {
+ rlmScheduledThreadPool.shutdown();
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ leaderOrFollowerTasks.clear();
+ closed = true;
+ }
+ }
+ }
+
+ private static class RLMScheduledThreadPool {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RLMScheduledThreadPool.class);
+ private final int poolSize;
+ private final ScheduledThreadPoolExecutor scheduledThreadPool;
+
+ public RLMScheduledThreadPool(int poolSize) {
+ this.poolSize = poolSize;
+ scheduledThreadPool = createPool();
+ }
+
+ private ScheduledThreadPoolExecutor createPool() {
+ ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(poolSize);
+ threadPool.setRemoveOnCancelPolicy(true);
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
+ threadPool.setThreadFactory(new ThreadFactory() {
+ private final AtomicInteger sequence = new AtomicInteger();
+
+ public Thread newThread(Runnable r) {
+ return KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r);
+ }
+ });
+
+ return threadPool;
+ }
+
+ public void resizePool(int size) {
+ LOGGER.info("Resizing pool from {} to {}", scheduledThreadPool.getCorePoolSize(), size);
+ scheduledThreadPool.setCorePoolSize(size);
+ }
+
+ public int poolSize() {
Review Comment:
This seems unused?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon merged pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon merged PR #13487:
URL: https://github.com/apache/kafka/pull/13487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] Hangleton commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "Hangleton (via GitHub)" <gi...@apache.org>.
Hangleton commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1166975428
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
Review Comment:
Why is the size of the remote index cache 1,024? It may not be sufficient for all use cases. Maybe, this could be configurable via a property.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
Review Comment:
( nit) `AccessController` has been tagged deprecated and for removal since JDK 17.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(RLMTask.class);
+ }
+
+ boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The copiedOffsetOption is OptionalLong.empty() initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition " + topicIdPartition + " can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void copyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed (high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
Review Comment:
Maybe we should log a message (warning?) if `copiedOffset + 1` < `log.logStartOffset()` in which case there may be a gap in the remote log? What if another replica (non-leader) still holds the data we are missing here?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(RLMTask.class);
+ }
+
+ boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The copiedOffsetOption is OptionalLong.empty() initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition " + topicIdPartition + " can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
Review Comment:
Hmm, not sure I understand the purpose of this statement. The invariant is always `leaderEpoch == leaderEpochVal` after line 440 with or without the conditional clause.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(RLMTask.class);
+ }
+
+ boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The copiedOffsetOption is OptionalLong.empty() initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition " + topicIdPartition + " can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void copyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed (high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ ListIterator<LogSegment> logSegmentsIter = sortedSegments.subList(0, activeSegIndex).listIterator();
+ while (logSegmentsIter.hasNext()) {
+ LogSegment segment = logSegmentsIter.next();
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ copyLogSegment(log, segment, getNextSegmentBaseOffset(activeSegBaseOffset, logSegmentsIter));
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private long getNextSegmentBaseOffset(long activeSegBaseOffset, ListIterator<LogSegment> logSegmentsIter) {
+ long nextSegmentBaseOffset;
+ if (logSegmentsIter.hasNext()) {
+ nextSegmentBaseOffset = logSegmentsIter.next().baseOffset();
+ logSegmentsIter.previous();
+ } else {
+ nextSegmentBaseOffset = activeSegBaseOffset;
+ }
+
+ return nextSegmentBaseOffset;
+ }
+
+ private void copyLogSegment(UnifiedLog log, LogSegment segment, long nextSegmentBaseOffset) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long endOffset = nextSegmentBaseOffset - 1;
+ File producerStateSnapshotFile = log.producerStateManager().fetchSnapshot(nextSegmentBaseOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextSegmentBaseOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextSegmentBaseOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerStateSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ copyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
Review Comment:
One potential problem with the current SPI is that the checked `InterruptedException` cannot be propagated from plugin implementations and intercepted here.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
Review Comment:
I wonder if we could rely on the topic-id mapping in the `MetadataCache` instead of maintaining the mapping locally. Is there any benefit in doing that here, versus using the `MetadataCache` as a source of truth?
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
Review Comment:
Maybe all the methods/contracts of the `RemoteLogManager` could be based on `TopicIdPartition`. It would then be the responsibility of the caller to perform the resolution of the topic ids if the Kafka API for which the invocation of the RLM is made does not support it.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match updated topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ // for testing
+ public RLMScheduledThreadPool rlmScheduledThreadPool() {
+ return rlmScheduledThreadPool;
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(RLMTask.class);
+ }
+
+ boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The copiedOffsetOption is OptionalLong.empty() initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition " + topicIdPartition + " can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
Review Comment:
Should the offset be reset if the leader epoch is the same as the one we already have? (it shouldn't hurt to query the highest remote offset multiple times in any case, so that is a minor observation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161286744
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
Review Comment:
This is needed for tests, which are added now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1158197887
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
+ epochEntries.forEach(entry =>
+ segmentLeaderEpochs.put(Integer.valueOf(entry.epoch), lang.Long.valueOf(entry.startOffset))
+ )
+
+ val remoteLogSegmentMetadata = new RemoteLogSegmentMetadata(id, segment.baseOffset, endOffset,
+ segment.largestTimestamp, brokerId, time.milliseconds(), segment.log.sizeInBytes(),
+ segmentLeaderEpochs)
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(remoteLogSegmentMetadata).get()
+
+ val leaderEpochsIndex = getLeaderEpochCheckpoint(log, startOffset = -1, nextOffset).readAsByteBuffer()
+ val segmentData = new LogSegmentData(logFile.toPath, toPathIfExists(segment.lazyOffsetIndex.get.file),
+ toPathIfExists(segment.lazyTimeIndex.get.file), Optional.ofNullable(toPathIfExists(segment.txnIndex.file)),
+ producerIdSnapshotFile.toPath, leaderEpochsIndex)
+ remoteLogStorageManager.copyLogSegmentData(remoteLogSegmentMetadata, segmentData)
+
+ val rlsmAfterSegmentCopy = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId)
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(rlsmAfterSegmentCopy).get()
+
+ copiedOffsetOption = Some(endOffset)
+ log.updateHighestOffsetInRemoteStorage(endOffset)
+ info(s"Copied $logFileName to remote storage with segment-id: ${rlsmAfterSegmentCopy.remoteLogSegmentId()}")
+ }
+ }
+ } else {
+ debug(s"Skipping copying segments, current read-offset:$copiedOffset, and LSO:$lso")
+ }
+ }
+ } catch {
+ case ex: Exception =>
+ if (!isCancelled()) {
+ error(s"Error occurred while copying log segments of partition: $tpId", ex)
+ }
+ }
+ }
+
+ private def toPathIfExists(file: File): Path = {
+ if (file.exists()) file.toPath else null
+ }
+
+ override def run(): Unit = {
+ if (isCancelled())
+ return
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote()
+ }
+ } catch {
+ case ex: InterruptedException =>
+ if (!isCancelled()) {
+ warn(s"Current thread for topic-partition-id $tpId is interrupted, this task won't be rescheduled. " +
+ s"Reason: ${ex.getMessage}")
+ }
+ case ex: Exception =>
+ if (!isCancelled()) {
+ warn(s"Current task for topic-partition $tpId received error but it will be scheduled. " +
+ s"Reason: ${ex.getMessage}")
+ }
+ }
+ }
+
+ override def toString: String = {
+ this.getClass.toString + s"[$tpId]"
+ }
+ }
+
+ def findHighestRemoteOffset(topicIdPartition: TopicIdPartition): Long = {
+ var offset: Optional[lang.Long] = Optional.empty()
+ fetchLog(topicIdPartition.topicPartition()).foreach { log =>
+ log.leaderEpochCache.foreach(cache => {
+ var epoch = cache.latestEpoch
+ while (!offset.isPresent && epoch.isPresent) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt)
+ epoch = cache.previousEpoch(epoch.getAsInt)
+ }
+ })
+ }
+ offset.orElse(-1L)
+ }
+
+
+ private def doHandleLeaderOrFollowerPartitions(topicPartition: TopicIdPartition,
+ convertToLeaderOrFollower: RLMTask => Unit): Unit = {
+ val rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition, (tp: TopicIdPartition) => {
+ val task = new RLMTask(tp)
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower(task)
+ info(s"Created a new task: $task and getting scheduled")
+ val future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS)
+ RLMTaskWithFuture(task, future)
+ })
+ convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)
Review Comment:
There is no race condition here. `rlmTaskWithFuture` will never be null as it will be created if it does not exist in `leaderOrFollowerTasks` map. It is expected that leader or follower switches can happen resulting leader epoch updates and `RLMTask` takes care of handling those updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1156860415
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,107 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
+import org.apache.kafka.common.internals.Topic
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
-import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
+import org.apache.kafka.server.log.remote.metadata.storage.{ClassLoaderAwareRemoteLogMetadataManager, TopicBasedRemoteLogMetadataManagerConfig}
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
Review Comment:
should we set `setContinueExistingPeriodicTasksAfterShutdownPolicy(false);` ?
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,107 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
+import org.apache.kafka.common.internals.Topic
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
-import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
+import org.apache.kafka.server.log.remote.metadata.storage.{ClassLoaderAwareRemoteLogMetadataManager, TopicBasedRemoteLogMetadataManagerConfig}
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
+ threadPool.setThreadFactory(new ThreadFactory {
+ private val sequence = new AtomicInteger()
+
+ override def newThread(r: Runnable): Thread = {
+ KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r)
+ }
+ })
+
+ threadPool
+ }
+
+ def resizePool(size: Int): Unit = {
+ info(s"Resizing pool from ${scheduledThreadPool.getCorePoolSize} to $size")
+ scheduledThreadPool.setCorePoolSize(size)
+ }
+
+ def poolSize(): Int = scheduledThreadPool.getCorePoolSize
+
+ def getIdlePercent(): Double = {
+ 1 - scheduledThreadPool.getActiveCount().asInstanceOf[Double] / scheduledThreadPool.getCorePoolSize.asInstanceOf[Double]
+ }
+
+ def scheduleWithFixedDelay(runnable: Runnable, initialDelay: Long, delay: Long,
+ timeUnit: TimeUnit): ScheduledFuture[_] = {
+ info(s"Scheduling runnable $runnable with initial delay: $initialDelay, fixed delay: $delay")
+ scheduledThreadPool.scheduleWithFixedDelay(runnable, initialDelay, delay, timeUnit)
+ }
+
+ def shutdown(): Boolean = {
+ info("Shutting down scheduled thread pool")
+ scheduledThreadPool.shutdownNow()
+ //waits for 2 mins to terminate the current tasks
Review Comment:
This comment is not needed.
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,107 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
+import org.apache.kafka.common.internals.Topic
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
-import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
+import org.apache.kafka.server.log.remote.metadata.storage.{ClassLoaderAwareRemoteLogMetadataManager, TopicBasedRemoteLogMetadataManagerConfig}
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
+ threadPool.setThreadFactory(new ThreadFactory {
+ private val sequence = new AtomicInteger()
+
+ override def newThread(r: Runnable): Thread = {
+ KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r)
+ }
+ })
+
+ threadPool
+ }
+
+ def resizePool(size: Int): Unit = {
+ info(s"Resizing pool from ${scheduledThreadPool.getCorePoolSize} to $size")
+ scheduledThreadPool.setCorePoolSize(size)
+ }
+
+ def poolSize(): Int = scheduledThreadPool.getCorePoolSize
+
+ def getIdlePercent(): Double = {
+ 1 - scheduledThreadPool.getActiveCount().asInstanceOf[Double] / scheduledThreadPool.getCorePoolSize.asInstanceOf[Double]
+ }
+
+ def scheduleWithFixedDelay(runnable: Runnable, initialDelay: Long, delay: Long,
+ timeUnit: TimeUnit): ScheduledFuture[_] = {
+ info(s"Scheduling runnable $runnable with initial delay: $initialDelay, fixed delay: $delay")
+ scheduledThreadPool.scheduleWithFixedDelay(runnable, initialDelay, delay, timeUnit)
+ }
+
+ def shutdown(): Boolean = {
+ info("Shutting down scheduled thread pool")
+ scheduledThreadPool.shutdownNow()
+ //waits for 2 mins to terminate the current tasks
+ scheduledThreadPool.awaitTermination(2, TimeUnit.MINUTES)
+ }
+}
+
+trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+}
+
/**
* This class is responsible for
- * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances.
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
* - receives any leader and follower replica events and partition stop events and act on them
- * - also provides APIs to fetch indexes, metadata about remote log segments.
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
*
* @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
* @param brokerId id of the current broker.
* @param logDir directory of Kafka log segments.
Review Comment:
Please remember to update the javadoc
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -520,6 +522,11 @@ class UnifiedLog(@volatile var logStartOffset: Long,
localLog.updateRecoveryPoint(offset)
}
}
+ def updateHighestOffsetInRemoteStorage(offset: Long): Unit = {
+ if (!remoteLogEnabled())
+ warn(s"Received update for highest offset with remote index as: $offset, the existing value: $highestOffsetInRemoteStorage")
Review Comment:
It's unclear what we tried to warn? Maybe something like this:
`warn(s"Unable to update the highest offset with remote index as: $offset, the existing value: $highestOffsetInRemoteStorage since remote storage is not enabled.")`
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +310,326 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
Review Comment:
Why should we declare `CancellableRunnable` twice in this class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161287718
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
Review Comment:
base offset changes in RLMTask, and it is logged now.
We plan to add a metric about tier offset lag. This is already implemented internally. We will raise a PR in future.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
Review Comment:
This is needed for tests which are added now.
##########
core/src/main/scala/kafka/log/UnifiedLog.scala:
##########
@@ -1231,10 +1238,10 @@ class UnifiedLog(@volatile var logStartOffset: Long,
}
remoteLogManager.get.findOffsetByTimestamp(topicPartition, targetTimestamp, logStartOffset, leaderEpochCache.get)
- } else None
+ } else Optional.empty()
Review Comment:
Earlier `remoteLogManager#findOffsetByTimestamp` was returning Scala `Option` but it returns `Optional` now after it is migrated to Java.
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ private void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+
+ }
+ );
+ convertToLeaderOrFollower.accept(rlmTaskWithFuture.rlmTask);
+ }
+
+ private static class RLMTaskWithFuture {
+
+ private final RLMTask rlmTask;
+ private final Future<?> future;
+
+ RLMTaskWithFuture(RLMTask rlmTask, Future<?> future) {
+ this.rlmTask = rlmTask;
+ this.future = future;
+ }
+
+ public void cancel() {
+ rlmTask.cancel();
+ try {
+ future.cancel(true);
+ } catch (Exception ex) {
+ LOGGER.error("Error occurred while canceling the task: {}", rlmTask, ex);
+ }
+ }
+
+ }
+
+ private static class InMemoryLeaderEpochCheckpoint implements LeaderEpochCheckpoint {
+ private List<EpochEntry> epochs = Collections.emptyList();
+
+ public void write(Collection<EpochEntry> epochs) {
+ this.epochs = new ArrayList<>(epochs);
+ }
+
+ public List<EpochEntry> read() {
+ return Collections.unmodifiableList(epochs);
+ }
+
+ public ByteBuffer readAsByteBuffer() throws IOException {
+ ByteArrayOutputStream stream = new ByteArrayOutputStream();
+ try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stream, StandardCharsets.UTF_8));) {
+ CheckpointFile.CheckpointWriteBuffer<EpochEntry> writeBuffer = new CheckpointFile.CheckpointWriteBuffer<>(writer, 0, LeaderEpochCheckpointFile.FORMATTER);
+ writeBuffer.write(epochs);
+ writer.flush();
+ }
+
+ return ByteBuffer.wrap(stream.toByteArray());
+ }
+ }
+
+ /**
+ * Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
+ */
+ public void close() {
+ synchronized (this) {
+ if (!closed) {
+ leaderOrFollowerTasks.values().forEach(RLMTaskWithFuture::cancel);
+ Utils.closeQuietly(remoteLogStorageManager, "RemoteLogStorageManager");
+ Utils.closeQuietly(remoteLogMetadataManager, "RemoteLogMetadataManager");
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
+ try {
+ rlmScheduledThreadPool.shutdown();
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ leaderOrFollowerTasks.clear();
+ closed = true;
+ }
+ }
+ }
+
+ private static class RLMScheduledThreadPool {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RLMScheduledThreadPool.class);
+ private final int poolSize;
+ private final ScheduledThreadPoolExecutor scheduledThreadPool;
+
+ public RLMScheduledThreadPool(int poolSize) {
+ this.poolSize = poolSize;
+ scheduledThreadPool = createPool();
+ }
+
+ private ScheduledThreadPoolExecutor createPool() {
+ ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(poolSize);
+ threadPool.setRemoveOnCancelPolicy(true);
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
+ threadPool.setThreadFactory(new ThreadFactory() {
+ private final AtomicInteger sequence = new AtomicInteger();
+
+ public Thread newThread(Runnable r) {
+ return KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r);
+ }
+ });
+
+ return threadPool;
+ }
+
+ public void resizePool(int size) {
+ LOGGER.info("Resizing pool from {} to {}", scheduledThreadPool.getCorePoolSize(), size);
+ scheduledThreadPool.setCorePoolSize(size);
+ }
+
+ public int poolSize() {
Review Comment:
This is used for dynamic config changes that we planned to add later, removed it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161286998
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
Review Comment:
Does not `copySegmentFinishedRlsm` represent what it is meant for instead of `finishedSegmentRlsm `?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13487:
URL: https://github.com/apache/kafka/pull/13487#issuecomment-1491884963
Thanks for the PR. I'll have an early review this weekend. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1157196981
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
+ epochEntries.forEach(entry =>
+ segmentLeaderEpochs.put(Integer.valueOf(entry.epoch), lang.Long.valueOf(entry.startOffset))
+ )
+
+ val remoteLogSegmentMetadata = new RemoteLogSegmentMetadata(id, segment.baseOffset, endOffset,
+ segment.largestTimestamp, brokerId, time.milliseconds(), segment.log.sizeInBytes(),
+ segmentLeaderEpochs)
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(remoteLogSegmentMetadata).get()
+
+ val leaderEpochsIndex = getLeaderEpochCheckpoint(log, startOffset = -1, nextOffset).readAsByteBuffer()
+ val segmentData = new LogSegmentData(logFile.toPath, toPathIfExists(segment.lazyOffsetIndex.get.file),
+ toPathIfExists(segment.lazyTimeIndex.get.file), Optional.ofNullable(toPathIfExists(segment.txnIndex.file)),
+ producerIdSnapshotFile.toPath, leaderEpochsIndex)
+ remoteLogStorageManager.copyLogSegmentData(remoteLogSegmentMetadata, segmentData)
+
+ val rlsmAfterSegmentCopy = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId)
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(rlsmAfterSegmentCopy).get()
+
+ copiedOffsetOption = Some(endOffset)
+ log.updateHighestOffsetInRemoteStorage(endOffset)
+ info(s"Copied $logFileName to remote storage with segment-id: ${rlsmAfterSegmentCopy.remoteLogSegmentId()}")
+ }
+ }
+ } else {
+ debug(s"Skipping copying segments, current read-offset:$copiedOffset, and LSO:$lso")
+ }
+ }
+ } catch {
+ case ex: Exception =>
+ if (!isCancelled()) {
+ error(s"Error occurred while copying log segments of partition: $tpId", ex)
+ }
+ }
+ }
+
+ private def toPathIfExists(file: File): Path = {
+ if (file.exists()) file.toPath else null
+ }
+
+ override def run(): Unit = {
+ if (isCancelled())
+ return
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote()
+ }
+ } catch {
+ case ex: InterruptedException =>
+ if (!isCancelled()) {
+ warn(s"Current thread for topic-partition-id $tpId is interrupted, this task won't be rescheduled. " +
+ s"Reason: ${ex.getMessage}")
+ }
+ case ex: Exception =>
+ if (!isCancelled()) {
+ warn(s"Current task for topic-partition $tpId received error but it will be scheduled. " +
+ s"Reason: ${ex.getMessage}")
+ }
+ }
+ }
+
+ override def toString: String = {
+ this.getClass.toString + s"[$tpId]"
+ }
+ }
+
+ def findHighestRemoteOffset(topicIdPartition: TopicIdPartition): Long = {
+ var offset: Optional[lang.Long] = Optional.empty()
+ fetchLog(topicIdPartition.topicPartition()).foreach { log =>
+ log.leaderEpochCache.foreach(cache => {
+ var epoch = cache.latestEpoch
+ while (!offset.isPresent && epoch.isPresent) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt)
+ epoch = cache.previousEpoch(epoch.getAsInt)
+ }
+ })
+ }
+ offset.orElse(-1L)
+ }
+
+
+ private def doHandleLeaderOrFollowerPartitions(topicPartition: TopicIdPartition,
+ convertToLeaderOrFollower: RLMTask => Unit): Unit = {
+ val rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition, (tp: TopicIdPartition) => {
+ val task = new RLMTask(tp)
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower(task)
+ info(s"Created a new task: $task and getting scheduled")
+ val future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS)
+ RLMTaskWithFuture(task, future)
+ })
+ convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)
Review Comment:
Will there be race condition when the topic partition is new inserted, so it'll set `convertToLeaderOrFollower` once before scheduling. And then the scheduled thread could be run immediately, later, run this line `convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)`, so it reset the leader epoch if it got updated.
Could we add a null condition check for `rlmTaskWithFuture` here? that is:
```
if (rlmTaskWithFuture != null) {
// not new created task, should convert it here
convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)
}
```
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
+ epochEntries.forEach(entry =>
+ segmentLeaderEpochs.put(Integer.valueOf(entry.epoch), lang.Long.valueOf(entry.startOffset))
+ )
+
+ val remoteLogSegmentMetadata = new RemoteLogSegmentMetadata(id, segment.baseOffset, endOffset,
+ segment.largestTimestamp, brokerId, time.milliseconds(), segment.log.sizeInBytes(),
+ segmentLeaderEpochs)
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(remoteLogSegmentMetadata).get()
+
+ val leaderEpochsIndex = getLeaderEpochCheckpoint(log, startOffset = -1, nextOffset).readAsByteBuffer()
+ val segmentData = new LogSegmentData(logFile.toPath, toPathIfExists(segment.lazyOffsetIndex.get.file),
+ toPathIfExists(segment.lazyTimeIndex.get.file), Optional.ofNullable(toPathIfExists(segment.txnIndex.file)),
+ producerIdSnapshotFile.toPath, leaderEpochsIndex)
+ remoteLogStorageManager.copyLogSegmentData(remoteLogSegmentMetadata, segmentData)
+
+ val rlsmAfterSegmentCopy = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId)
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(rlsmAfterSegmentCopy).get()
+
+ copiedOffsetOption = Some(endOffset)
+ log.updateHighestOffsetInRemoteStorage(endOffset)
+ info(s"Copied $logFileName to remote storage with segment-id: ${rlsmAfterSegmentCopy.remoteLogSegmentId()}")
+ }
+ }
+ } else {
+ debug(s"Skipping copying segments, current read-offset:$copiedOffset, and LSO:$lso")
+ }
+ }
+ } catch {
+ case ex: Exception =>
+ if (!isCancelled()) {
+ error(s"Error occurred while copying log segments of partition: $tpId", ex)
+ }
+ }
+ }
+
+ private def toPathIfExists(file: File): Path = {
+ if (file.exists()) file.toPath else null
+ }
+
+ override def run(): Unit = {
+ if (isCancelled())
+ return
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote()
+ }
+ } catch {
+ case ex: InterruptedException =>
+ if (!isCancelled()) {
+ warn(s"Current thread for topic-partition-id $tpId is interrupted, this task won't be rescheduled. " +
+ s"Reason: ${ex.getMessage}")
+ }
+ case ex: Exception =>
+ if (!isCancelled()) {
+ warn(s"Current task for topic-partition $tpId received error but it will be scheduled. " +
+ s"Reason: ${ex.getMessage}")
+ }
+ }
+ }
+
+ override def toString: String = {
+ this.getClass.toString + s"[$tpId]"
+ }
+ }
+
+ def findHighestRemoteOffset(topicIdPartition: TopicIdPartition): Long = {
+ var offset: Optional[lang.Long] = Optional.empty()
+ fetchLog(topicIdPartition.topicPartition()).foreach { log =>
+ log.leaderEpochCache.foreach(cache => {
+ var epoch = cache.latestEpoch
+ while (!offset.isPresent && epoch.isPresent) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt)
+ epoch = cache.previousEpoch(epoch.getAsInt)
+ }
+ })
+ }
+ offset.orElse(-1L)
+ }
+
+
+ private def doHandleLeaderOrFollowerPartitions(topicPartition: TopicIdPartition,
+ convertToLeaderOrFollower: RLMTask => Unit): Unit = {
+ val rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition, (tp: TopicIdPartition) => {
+ val task = new RLMTask(tp)
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower(task)
+ info(s"Created a new task: $task and getting scheduled")
+ val future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS)
+ RLMTaskWithFuture(task, future)
+ })
+ convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)
Review Comment:
Will there be race condition when the topic partition is new inserted, so it'll set `convertToLeaderOrFollower` once before scheduling. And then the scheduled thread could be run immediately, later, run this line `convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)`, so it reset the leader epoch if it got updated.
Could we add a null condition check for `rlmTaskWithFuture` here? that is:
```
if (rlmTaskWithFuture != null) {
// not new created task, should convert it here
convertToLeaderOrFollower(rlmTaskWithFuture.rlmTask)
}
```
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1158205451
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
Review Comment:
The existing `Integer` represents that, adding `lang` is redundant. We had to add `lang` to Long as it conflicts with scala's `Long`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1156812334
##########
storage/src/main/java/org/apache/kafka/storage/internals/epoch/LeaderEpochFileCache.java:
##########
@@ -359,6 +361,27 @@ public OptionalInt epochForOffset(long offset) {
}
}
+ public LeaderEpochFileCache writeTo(LeaderEpochCheckpoint leaderEpochCheckpoint) {
+ lock.readLock().lock();
+ try {
+ leaderEpochCheckpoint.write(epochEntries());
+ return new LeaderEpochFileCache(topicPartition, leaderEpochCheckpoint);
+ } finally {
+ lock.readLock().unlock();
+ }
+ }
+
+ public OptionalInt findPreviousEpoch(int epoch) {
Review Comment:
Good catch! This method is already available as `previousEpoch(int epoch)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "junrao (via GitHub)" <gi...@apache.org>.
junrao commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1158975829
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,97 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false)
+ threadPool.setThreadFactory(new ThreadFactory {
+ private val sequence = new AtomicInteger()
+
+ override def newThread(r: Runnable): Thread = {
+ KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r)
+ }
+ })
+
+ threadPool
+ }
+
+ def resizePool(size: Int): Unit = {
+ info(s"Resizing pool from ${scheduledThreadPool.getCorePoolSize} to $size")
+ scheduledThreadPool.setCorePoolSize(size)
+ }
+
+ def poolSize(): Int = scheduledThreadPool.getCorePoolSize
+
+ def getIdlePercent(): Double = {
+ 1 - scheduledThreadPool.getActiveCount().asInstanceOf[Double] / scheduledThreadPool.getCorePoolSize.asInstanceOf[Double]
+ }
+
+ def scheduleWithFixedDelay(runnable: Runnable, initialDelay: Long, delay: Long,
+ timeUnit: TimeUnit): ScheduledFuture[_] = {
+ info(s"Scheduling runnable $runnable with initial delay: $initialDelay, fixed delay: $delay")
+ scheduledThreadPool.scheduleWithFixedDelay(runnable, initialDelay, delay, timeUnit)
+ }
+
+ def shutdown(): Boolean = {
+ info("Shutting down scheduled thread pool")
+ scheduledThreadPool.shutdownNow()
+ //waits for 2 mins to terminate the current tasks
+ scheduledThreadPool.awaitTermination(2, TimeUnit.MINUTES)
+ }
+}
+
/**
* This class is responsible for
- * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances.
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
* - receives any leader and follower replica events and partition stop events and act on them
- * - also provides APIs to fetch indexes, metadata about remote log segments.
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
*
* @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
* @param brokerId id of the current broker.
* @param logDir directory of Kafka log segments.
+ * @param time Time instance.
Review Comment:
Since this is a relatively new class and we are in the process of migrating scala classes to java, would it be better to write this class in java but still keep it in the core module? This way, we don't need to write the new code twice, once in scala and another in java.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1160112094
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -17,39 +17,97 @@
package kafka.log.remote
import kafka.cluster.Partition
+import kafka.log.UnifiedLog
import kafka.server.KafkaConfig
import kafka.utils.Logging
import org.apache.kafka.common._
import org.apache.kafka.common.record.FileRecords.TimestampAndOffset
import org.apache.kafka.common.record.{RecordBatch, RemoteLogInputStream}
-import org.apache.kafka.common.utils.{ChildFirstClassLoader, Utils}
+import org.apache.kafka.common.utils.{ChildFirstClassLoader, KafkaThread, Time, Utils}
+import org.apache.kafka.server.common.CheckpointFile.CheckpointWriteBuffer
import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager
import org.apache.kafka.server.log.remote.storage._
+import org.apache.kafka.storage.internals.checkpoint.{LeaderEpochCheckpoint, LeaderEpochCheckpointFile}
import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache
+import org.apache.kafka.storage.internals.log.EpochEntry
-import java.io.{Closeable, InputStream}
+import java.io._
+import java.nio.ByteBuffer
+import java.nio.charset.StandardCharsets
+import java.nio.file.Path
import java.security.{AccessController, PrivilegedAction}
-import java.util
import java.util.Optional
-import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
+import java.util.concurrent._
+import java.util.concurrent.atomic.AtomicInteger
+import java.{lang, util}
+import scala.collection.Searching._
import scala.collection.Set
import scala.jdk.CollectionConverters._
+class RLMScheduledThreadPool(poolSize: Int) extends Logging {
+
+ private val scheduledThreadPool: ScheduledThreadPoolExecutor = {
+ val threadPool = new ScheduledThreadPoolExecutor(poolSize)
+ threadPool.setRemoveOnCancelPolicy(true)
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false)
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false)
+ threadPool.setThreadFactory(new ThreadFactory {
+ private val sequence = new AtomicInteger()
+
+ override def newThread(r: Runnable): Thread = {
+ KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r)
+ }
+ })
+
+ threadPool
+ }
+
+ def resizePool(size: Int): Unit = {
+ info(s"Resizing pool from ${scheduledThreadPool.getCorePoolSize} to $size")
+ scheduledThreadPool.setCorePoolSize(size)
+ }
+
+ def poolSize(): Int = scheduledThreadPool.getCorePoolSize
+
+ def getIdlePercent(): Double = {
+ 1 - scheduledThreadPool.getActiveCount().asInstanceOf[Double] / scheduledThreadPool.getCorePoolSize.asInstanceOf[Double]
+ }
+
+ def scheduleWithFixedDelay(runnable: Runnable, initialDelay: Long, delay: Long,
+ timeUnit: TimeUnit): ScheduledFuture[_] = {
+ info(s"Scheduling runnable $runnable with initial delay: $initialDelay, fixed delay: $delay")
+ scheduledThreadPool.scheduleWithFixedDelay(runnable, initialDelay, delay, timeUnit)
+ }
+
+ def shutdown(): Boolean = {
+ info("Shutting down scheduled thread pool")
+ scheduledThreadPool.shutdownNow()
+ //waits for 2 mins to terminate the current tasks
+ scheduledThreadPool.awaitTermination(2, TimeUnit.MINUTES)
+ }
+}
+
/**
* This class is responsible for
- * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances.
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
* - receives any leader and follower replica events and partition stop events and act on them
- * - also provides APIs to fetch indexes, metadata about remote log segments.
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
*
* @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
* @param brokerId id of the current broker.
* @param logDir directory of Kafka log segments.
+ * @param time Time instance.
Review Comment:
This was an existing code in Scala in my other branch with a few changes. But your suggestion avoids the review again during Java migration later. I did the Java migration as you suggested and pushed it to the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #13487:
URL: https://github.com/apache/kafka/pull/13487#issuecomment-1504387988
Failed tests are unrelated:
```
Build / JDK 17 and Scala 2.13 / kafka.admin.TopicCommandIntegrationTest.testDescribeAndListTopicsWithoutInternalTopics(String).quorum=zk
Build / JDK 17 and Scala 2.13 / kafka.api.TransactionsTest.testFailureToFenceEpoch(String).quorum=kraft
Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted
Build / JDK 11 and Scala 2.13 / kafka.security.authorizer.AuthorizerTest.testSuperUserHasAccess(String).quorum=kraft
Build / JDK 11 and Scala 2.13 / kafka.server.DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable()
Build / JDK 8 and Scala 2.12 / kafka.security.authorizer.AuthorizerTest.testNoAclFoundOverride(String).quorum=kraft
Build / JDK 8 and Scala 2.12 / kafka.security.authorizer.AuthorizerTest.testSuperUserWithCustomPrincipalHasAccess(String).quorum=kraft
Build / JDK 8 and Scala 2.12 / kafka.security.authorizer.AuthorizerTest.testDeleteAclOnWildcardResource(String).quorum=kraft
```
Since this PR blocks other following PRs, I'm going to merge it soon. We can have a follow-up PR to address any new comments if any. Thank you.
cc @junrao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] Hangleton commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "Hangleton (via GitHub)" <gi...@apache.org>.
Hangleton commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1167007616
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,719 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.InMemoryLeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
Review Comment:
(nit) `AccessController` has been tagged deprecated and for removal since JDK 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161670474
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
Review Comment:
I think the default code format is about having space before and after ':'. I see the same in all other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1155880909
##########
storage/src/main/java/org/apache/kafka/storage/internals/epoch/LeaderEpochFileCache.java:
##########
@@ -359,6 +361,27 @@ public OptionalInt epochForOffset(long offset) {
}
}
+ public LeaderEpochFileCache writeTo(LeaderEpochCheckpoint leaderEpochCheckpoint) {
+ lock.readLock().lock();
+ try {
+ leaderEpochCheckpoint.write(epochEntries());
+ return new LeaderEpochFileCache(topicPartition, leaderEpochCheckpoint);
+ } finally {
+ lock.readLock().unlock();
+ }
+ }
+
+ public OptionalInt findPreviousEpoch(int epoch) {
Review Comment:
Any reason we don't use the existing `previousEpoch` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1158203473
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
Review Comment:
LastStableOffset already represents that, will update the javadoc comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1158198885
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -143,24 +216,44 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
topicIds: util.Map[String, Uuid]): Unit = {
debug(s"Received leadership changes for leaders: $partitionsBecomeLeader and followers: $partitionsBecomeFollower")
- // Partitions logs are available when this callback is invoked.
- // Compact topics and internal topics are filtered here as they are not supported with tiered storage.
- def filterPartitions(partitions: Set[Partition]): Set[TopicIdPartition] = {
+ def filterPartitions(partitions: Set[Partition]): Set[Partition] = {
// We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
partitions.filter(partition => partition.log.exists(log => log.remoteLogEnabled()))
- .map(partition => new TopicIdPartition(topicIds.get(partition.topic), partition.topicPartition))
}
- val followerTopicPartitions = filterPartitions(partitionsBecomeFollower)
- val leaderTopicPartitions = filterPartitions(partitionsBecomeLeader)
- debug(s"Effective topic partitions after filtering compact and internal topics, leaders: $leaderTopicPartitions " +
- s"and followers: $followerTopicPartitions")
+ val leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .map(p => new TopicIdPartition(topicIds.get(p.topic), p.topicPartition) -> p.getLeaderEpoch).toMap
+ val leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet
- if (leaderTopicPartitions.nonEmpty || followerTopicPartitions.nonEmpty) {
- leaderTopicPartitions.foreach(x => topicPartitionIds.put(x.topicPartition(), x.topicId()))
- followerTopicPartitions.foreach(x => topicPartitionIds.put(x.topicPartition(), x.topicId()))
+ val followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p => new TopicIdPartition(topicIds.get(p.topic), p.topicPartition))
+
+ def cacheTopicPartitionIds(topicIdPartition: TopicIdPartition): Unit = {
+ val previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId())
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ warn(s"Previous cached topic id $previousTopicId for ${topicIdPartition.topicPartition()} does " +
Review Comment:
Changed it to info level, which may be useful for debugging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1157152232
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -143,24 +216,44 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
topicIds: util.Map[String, Uuid]): Unit = {
debug(s"Received leadership changes for leaders: $partitionsBecomeLeader and followers: $partitionsBecomeFollower")
- // Partitions logs are available when this callback is invoked.
- // Compact topics and internal topics are filtered here as they are not supported with tiered storage.
- def filterPartitions(partitions: Set[Partition]): Set[TopicIdPartition] = {
+ def filterPartitions(partitions: Set[Partition]): Set[Partition] = {
// We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
partitions.filter(partition => partition.log.exists(log => log.remoteLogEnabled()))
- .map(partition => new TopicIdPartition(topicIds.get(partition.topic), partition.topicPartition))
}
- val followerTopicPartitions = filterPartitions(partitionsBecomeFollower)
- val leaderTopicPartitions = filterPartitions(partitionsBecomeLeader)
- debug(s"Effective topic partitions after filtering compact and internal topics, leaders: $leaderTopicPartitions " +
- s"and followers: $followerTopicPartitions")
+ val leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .map(p => new TopicIdPartition(topicIds.get(p.topic), p.topicPartition) -> p.getLeaderEpoch).toMap
+ val leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet
- if (leaderTopicPartitions.nonEmpty || followerTopicPartitions.nonEmpty) {
- leaderTopicPartitions.foreach(x => topicPartitionIds.put(x.topicPartition(), x.topicId()))
- followerTopicPartitions.foreach(x => topicPartitionIds.put(x.topicPartition(), x.topicId()))
+ val followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p => new TopicIdPartition(topicIds.get(p.topic), p.topicPartition))
+
+ def cacheTopicPartitionIds(topicIdPartition: TopicIdPartition): Unit = {
+ val previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId())
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ warn(s"Previous cached topic id $previousTopicId for ${topicIdPartition.topicPartition()} does " +
Review Comment:
This warning message seems not necessary IMO. Users cannot do anything when getting this info, right?
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
Review Comment:
Where do we set the offset with `min of high-watermark or stable-offset`?
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
Review Comment:
Should we use `lang.Integer` here?
##########
core/src/main/scala/kafka/log/remote/RemoteLogManager.scala:
##########
@@ -272,15 +364,270 @@ class RemoteLogManager(rlmConfig: RemoteLogManagerConfig,
None
}
+ trait CancellableRunnable extends Runnable {
+ @volatile private var cancelled = false
+
+ def cancel(): Unit = {
+ cancelled = true
+ }
+
+ def isCancelled(): Boolean = {
+ cancelled
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+ private[remote] def getLeaderEpochCheckpoint(log: UnifiedLog, startOffset: Long, endOffset: Long): InMemoryLeaderEpochCheckpoint = {
+ val checkpoint = new InMemoryLeaderEpochCheckpoint()
+ log.leaderEpochCache
+ .map(cache => cache.writeTo(checkpoint))
+ .foreach { x =>
+ if (startOffset >= 0) {
+ x.truncateFromStart(startOffset)
+ }
+ x.truncateFromEnd(endOffset)
+ }
+ checkpoint
+ }
+
+ private[remote] class RLMTask(tpId: TopicIdPartition) extends CancellableRunnable with Logging {
+ this.logIdent = s"[RemoteLogManager=$brokerId partition=$tpId] "
+ @volatile private var leaderEpoch: Int = -1
+
+ private def isLeader(): Boolean = leaderEpoch >= 0
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ @volatile private var copiedOffsetOption: Option[Long] = None
+
+ def convertToLeader(leaderEpochVal: Int): Unit = {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException(s"leaderEpoch value for topic partition $tpId can not be negative")
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = None
+ }
+
+ def convertToFollower(): Unit = {
+ leaderEpoch = -1
+ }
+
+ def handleCopyLogSegmentsToRemote(): Unit = {
+ if (isCancelled())
+ return
+
+ def maybeUpdateReadOffset(): Unit = {
+ if (copiedOffsetOption.isEmpty) {
+ info(s"Find the highest remote offset for partition: $tpId after becoming leader, leaderEpoch: $leaderEpoch")
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = Some(findHighestRemoteOffset(tpId))
+ }
+ }
+
+ try {
+ maybeUpdateReadOffset()
+ val copiedOffset = copiedOffsetOption.get
+ fetchLog(tpId.topicPartition()).foreach { log =>
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ val lso = log.lastStableOffset
+ if (lso < 0) {
+ warn(s"lastStableOffset for partition $tpId is $lso, which should not be negative.")
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the min of high-watermark or stable-offset as remote storage should contain
+ // only committed/acked messages
+ val toOffset = lso
+ debug(s"Checking for segments to copy, copiedOffset: $copiedOffset and toOffset: $toOffset")
+ val activeSegBaseOffset = log.activeSegment.baseOffset
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ val fromOffset = Math.max(copiedOffset + 1, log.logStartOffset)
+ val sortedSegments = log.logSegments(fromOffset, toOffset).toSeq.sortBy(_.baseOffset)
+ val activeSegIndex: Int = sortedSegments.map(x => x.baseOffset).search(activeSegBaseOffset) match {
+ case Found(x) => x
+ case InsertionPoint(y) => y - 1
+ }
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ debug(s"No segments found to be copied for partition $tpId with copiedOffset: $copiedOffset and " +
+ s"active segment's base-offset: $activeSegBaseOffset")
+ } else {
+ sortedSegments.slice(0, activeSegIndex).foreach { segment =>
+ if (isCancelled() || !isLeader()) {
+ info(s"Skipping copying log segments as the current task state is changed, cancelled: " +
+ s"${isCancelled()} leader:${isLeader()}")
+ return
+ }
+
+ val logFile = segment.log.file()
+ val logFileName = logFile.getName
+
+ info(s"Copying $logFileName to remote storage.")
+ val id = new RemoteLogSegmentId(tpId, Uuid.randomUuid())
+
+ val nextOffset = segment.readNextOffset
+ val endOffset = nextOffset - 1
+ val producerIdSnapshotFile: File = log.producerStateManager.fetchSnapshot(nextOffset).orElse(null)
+
+ val epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset, nextOffset).read()
+ val segmentLeaderEpochs: util.Map[Integer, lang.Long] = new util.HashMap[Integer, lang.Long](epochEntries.size())
+ epochEntries.forEach(entry =>
+ segmentLeaderEpochs.put(Integer.valueOf(entry.epoch), lang.Long.valueOf(entry.startOffset))
+ )
+
+ val remoteLogSegmentMetadata = new RemoteLogSegmentMetadata(id, segment.baseOffset, endOffset,
Review Comment:
could we rename this variable to a meaningful name like `copySegmentStartedMetadata`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] satishd commented on a diff in pull request #13487: KAFKA-9550 Copying log segments to tiered storage in RemoteLogManager
Posted by "satishd (via GitHub)" <gi...@apache.org>.
satishd commented on code in PR #13487:
URL: https://github.com/apache/kafka/pull/13487#discussion_r1161287264
##########
core/src/main/java/kafka/log/remote/RemoteLogManager.java:
##########
@@ -0,0 +1,749 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package kafka.log.remote;
+
+import kafka.cluster.Partition;
+import kafka.log.LogSegment;
+import kafka.log.UnifiedLog;
+import kafka.server.KafkaConfig;
+import org.apache.kafka.common.KafkaException;
+import org.apache.kafka.common.TopicIdPartition;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.record.FileRecords;
+import org.apache.kafka.common.record.Record;
+import org.apache.kafka.common.record.RecordBatch;
+import org.apache.kafka.common.record.RemoteLogInputStream;
+import org.apache.kafka.common.utils.ChildFirstClassLoader;
+import org.apache.kafka.common.utils.KafkaThread;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.server.common.CheckpointFile;
+import org.apache.kafka.server.log.remote.metadata.storage.ClassLoaderAwareRemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.ClassLoaderAwareRemoteStorageManager;
+import org.apache.kafka.server.log.remote.storage.LogSegmentData;
+import org.apache.kafka.server.log.remote.storage.RemoteLogManagerConfig;
+import org.apache.kafka.server.log.remote.storage.RemoteLogMetadataManager;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentId;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadata;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentMetadataUpdate;
+import org.apache.kafka.server.log.remote.storage.RemoteLogSegmentState;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageException;
+import org.apache.kafka.server.log.remote.storage.RemoteStorageManager;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpoint;
+import org.apache.kafka.storage.internals.checkpoint.LeaderEpochCheckpointFile;
+import org.apache.kafka.storage.internals.epoch.LeaderEpochFileCache;
+import org.apache.kafka.storage.internals.log.EpochEntry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import scala.Option;
+import scala.collection.JavaConverters;
+
+import java.io.BufferedWriter;
+import java.io.ByteArrayOutputStream;
+import java.io.Closeable;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStreamWriter;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Path;
+import java.security.AccessController;
+import java.security.PrivilegedAction;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.OptionalInt;
+import java.util.OptionalLong;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * This class is responsible for
+ * - initializing `RemoteStorageManager` and `RemoteLogMetadataManager` instances
+ * - receives any leader and follower replica events and partition stop events and act on them
+ * - also provides APIs to fetch indexes, metadata about remote log segments
+ * - copying log segments to remote storage
+ */
+public class RemoteLogManager implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RemoteLogManager.class);
+
+ private final RemoteLogManagerConfig rlmConfig;
+ private final int brokerId;
+ private final String logDir;
+ private final Time time;
+ private final Function<TopicPartition, Optional<UnifiedLog>> fetchLog;
+
+ private final RemoteStorageManager remoteLogStorageManager;
+
+ private final RemoteLogMetadataManager remoteLogMetadataManager;
+
+ private final RemoteIndexCache indexCache;
+
+ private final RLMScheduledThreadPool rlmScheduledThreadPool;
+
+ private final long delayInMs;
+
+ private final ConcurrentHashMap<TopicIdPartition, RLMTaskWithFuture> leaderOrFollowerTasks = new ConcurrentHashMap<>();
+
+ // topic ids that are received on leadership changes, this map is cleared on stop partitions
+ private final ConcurrentMap<TopicPartition, Uuid> topicPartitionIds = new ConcurrentHashMap<>();
+
+ private boolean closed = false;
+
+ /**
+ * Creates RemoteLogManager instance with the given arguments.
+ *
+ * @param rlmConfig Configuration required for remote logging subsystem(tiered storage) at the broker level.
+ * @param brokerId id of the current broker.
+ * @param logDir directory of Kafka log segments.
+ * @param time Time instance.
+ * @param fetchLog function to get UnifiedLog instance for a given topic.
+ * @throws Exception
+ */
+ public RemoteLogManager(RemoteLogManagerConfig rlmConfig,
+ int brokerId,
+ String logDir,
+ Time time,
+ Function<TopicPartition, Optional<UnifiedLog>> fetchLog) throws Exception {
+
+ this.rlmConfig = rlmConfig;
+ this.brokerId = brokerId;
+ this.logDir = logDir;
+ this.time = time;
+ this.fetchLog = fetchLog;
+
+ remoteLogStorageManager = createRemoteStorageManager();
+ remoteLogMetadataManager = createRemoteLogMetadataManager();
+ indexCache = new RemoteIndexCache(1024, remoteLogStorageManager, logDir);
+ delayInMs = rlmConfig.remoteLogManagerTaskIntervalMs();
+ rlmScheduledThreadPool = new RLMScheduledThreadPool(rlmConfig.remoteLogManagerThreadPoolSize());
+ }
+
+ private <T> T createDelegate(ClassLoader classLoader, String className) {
+ try {
+ return (T) classLoader.loadClass(className)
+ .getDeclaredConstructor().newInstance();
+ } catch (InstantiationException | IllegalAccessException | InvocationTargetException | NoSuchMethodException |
+ ClassNotFoundException e) {
+ throw new KafkaException(e);
+ }
+ }
+
+ RemoteStorageManager createRemoteStorageManager() throws Exception {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteStorageManager>() {
+ private final String classPath = rlmConfig.remoteStorageManagerClassPath();
+
+ public RemoteStorageManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ChildFirstClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteStorageManager delegate = createDelegate(classLoader, rlmConfig.remoteStorageManagerClassName());
+ return new ClassLoaderAwareRemoteStorageManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteStorageManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRSM() {
+ final Map<String, Object> rsmProps = new HashMap<>(rlmConfig.remoteStorageManagerProps());
+ rsmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ remoteLogStorageManager.configure(rsmProps);
+ }
+
+ RemoteLogMetadataManager createRemoteLogMetadataManager() {
+ return AccessController.doPrivileged(new PrivilegedAction<RemoteLogMetadataManager>() {
+ private String classPath = rlmConfig.remoteLogMetadataManagerClassPath();
+
+ public RemoteLogMetadataManager run() {
+ if (classPath != null && !classPath.trim().isEmpty()) {
+ ClassLoader classLoader = new ChildFirstClassLoader(classPath, this.getClass().getClassLoader());
+ RemoteLogMetadataManager delegate = createDelegate(classLoader, rlmConfig.remoteLogMetadataManagerClassName());
+ return new ClassLoaderAwareRemoteLogMetadataManager(delegate, classLoader);
+ } else {
+ return createDelegate(this.getClass().getClassLoader(), rlmConfig.remoteLogMetadataManagerClassName());
+ }
+ }
+ });
+ }
+
+ private void configureRLMM() {
+ final Map<String, Object> rlmmProps = new HashMap<>(rlmConfig.remoteLogMetadataManagerProps());
+
+ rlmmProps.put(KafkaConfig.BrokerIdProp(), brokerId);
+ rlmmProps.put(KafkaConfig.LogDirProp(), logDir);
+ remoteLogMetadataManager.configure(rlmmProps);
+ }
+
+ public void startup() {
+ // Initialize and configure RSM and RLMM. This will start RSM, RLMM resources which may need to start resources
+ // in connecting to the brokers or remote storages.
+ configureRSM();
+ configureRLMM();
+ }
+
+ public RemoteStorageManager storageManager() {
+ return remoteLogStorageManager;
+ }
+
+
+ private Stream<Partition> filterPartitions(Set<Partition> partitions) {
+ // We are not specifically checking for internal topics etc here as `log.remoteLogEnabled()` already handles that.
+ return partitions.stream().filter(partition -> partition.log().exists(UnifiedLog::remoteLogEnabled));
+ }
+
+ private void cacheTopicPartitionIds(TopicIdPartition topicIdPartition) {
+ Uuid previousTopicId = topicPartitionIds.put(topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ if (previousTopicId != null && previousTopicId != topicIdPartition.topicId()) {
+ LOGGER.info("Previous cached topic id {} for {} does not match update topic id {}",
+ previousTopicId, topicIdPartition.topicPartition(), topicIdPartition.topicId());
+ }
+ }
+
+ /**
+ * Callback to receive any leadership changes for the topic partitions assigned to this broker. If there are no
+ * existing tasks for a given topic partition then it will assign new leader or follower task else it will convert the
+ * task to respective target state(leader or follower).
+ *
+ * @param partitionsBecomeLeader partitions that have become leaders on this broker.
+ * @param partitionsBecomeFollower partitions that have become followers on this broker.
+ * @param topicIds topic name to topic id mappings.
+ */
+ public void onLeadershipChange(Set<Partition> partitionsBecomeLeader,
+ Set<Partition> partitionsBecomeFollower,
+ Map<String, Uuid> topicIds) {
+ LOGGER.debug("Received leadership changes for leaders: {} and followers: {}", partitionsBecomeLeader, partitionsBecomeFollower);
+
+
+ Map<TopicIdPartition, Integer> leaderPartitionsWithLeaderEpoch = filterPartitions(partitionsBecomeLeader)
+ .collect(Collectors.toMap(
+ partition -> new TopicIdPartition(topicIds.get(partition.topic()), partition.topicPartition()),
+ partition -> partition.getLeaderEpoch()));
+ Set<TopicIdPartition> leaderPartitions = leaderPartitionsWithLeaderEpoch.keySet();
+
+ Set<TopicIdPartition> followerPartitions = filterPartitions(partitionsBecomeFollower)
+ .map(p -> new TopicIdPartition(topicIds.get(p.topic()), p.topicPartition())).collect(Collectors.toSet());
+
+
+ if (!leaderPartitions.isEmpty() || !followerPartitions.isEmpty()) {
+ LOGGER.debug("Effective topic partitions after filtering compact and internal topics, leaders: {} and followers: {}",
+ leaderPartitions, followerPartitions);
+
+ leaderPartitions.forEach(this::cacheTopicPartitionIds);
+ followerPartitions.forEach(this::cacheTopicPartitionIds);
+
+ remoteLogMetadataManager.onPartitionLeadershipChanges(leaderPartitions, followerPartitions);
+ followerPartitions.forEach(topicIdPartition ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition, rlmTask -> rlmTask.convertToFollower()));
+
+ leaderPartitionsWithLeaderEpoch.forEach((topicIdPartition, leaderEpoch) ->
+ doHandleLeaderOrFollowerPartitions(topicIdPartition,
+ rlmTask -> rlmTask.convertToLeader(leaderEpoch)));
+ }
+ }
+
+ /**
+ * Deletes the internal topic partition info if delete flag is set as true.
+ *
+ * @param topicPartition topic partition to be stopped.
+ * @param delete flag to indicate whether the given topic partitions to be deleted or not.
+ */
+ public void stopPartitions(TopicPartition topicPartition, boolean delete) {
+ if (delete) {
+ // Delete from internal datastructures only if it is to be deleted.
+ Uuid topicIdPartition = topicPartitionIds.remove(topicPartition);
+ LOGGER.debug("Removed partition: {} from topicPartitionIds", topicIdPartition);
+ }
+ }
+
+ public Optional<RemoteLogSegmentMetadata> fetchRemoteLogSegmentMetadata(TopicPartition topicPartition,
+ int epochForOffset,
+ long offset) throws RemoteStorageException {
+ Uuid topicId = topicPartitionIds.get(topicPartition);
+
+ if (topicId == null) {
+ throw new KafkaException("No topic id registered for topic partition: " + topicPartition);
+ }
+ return remoteLogMetadataManager.remoteLogSegmentMetadata(new TopicIdPartition(topicId, topicPartition), epochForOffset, offset);
+ }
+
+ private Optional<FileRecords.TimestampAndOffset> lookupTimestamp(RemoteLogSegmentMetadata rlsMetadata, long timestamp, long startingOffset)
+ throws RemoteStorageException, IOException {
+ int startPos = indexCache.lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+
+ InputStream remoteSegInputStream = null;
+ try {
+ // Search forward for the position of the last offset that is greater than or equal to the startingOffset
+ remoteSegInputStream = remoteLogStorageManager.fetchLogSegment(rlsMetadata, startPos);
+ RemoteLogInputStream remoteLogInputStream = new RemoteLogInputStream(remoteSegInputStream);
+
+ while (true) {
+ RecordBatch batch = remoteLogInputStream.nextBatch();
+ if (batch == null) break;
+ if (batch.maxTimestamp() >= timestamp && batch.lastOffset() >= startingOffset) {
+ for (Record record : batch) {
+ if (record.timestamp() >= timestamp && record.offset() >= startingOffset)
+ return Optional.of(new FileRecords.TimestampAndOffset(record.timestamp(), record.offset(), maybeLeaderEpoch(batch.partitionLeaderEpoch())));
+ }
+ }
+ }
+
+ return Optional.empty();
+ } finally {
+ Utils.closeQuietly(remoteSegInputStream, "RemoteLogSegmentInputStream");
+ }
+ }
+
+ private Optional<Integer> maybeLeaderEpoch(int leaderEpoch) {
+ return leaderEpoch == RecordBatch.NO_PARTITION_LEADER_EPOCH ? Optional.empty() : Optional.of(leaderEpoch);
+ }
+
+ /**
+ * Search the message offset in the remote storage based on timestamp and offset.
+ * <p>
+ * This method returns an option of TimestampOffset. The returned value is determined using the following ordered list of rules:
+ * <p>
+ * - If there are no messages in the remote storage, return None
+ * - If all the messages in the remote storage have smaller offsets, return None
+ * - If all the messages in the remote storage have smaller timestamps, return None
+ * - Otherwise, return an option of TimestampOffset. The offset is the offset of the first message whose timestamp
+ * is greater than or equals to the target timestamp and whose offset is greater than or equals to the startingOffset.
+ *
+ * @param tp topic partition in which the offset to be found.
+ * @param timestamp The timestamp to search for.
+ * @param startingOffset The starting offset to search.
+ * @param leaderEpochCache LeaderEpochFileCache of the topic partition.
+ * @return the timestamp and offset of the first message that meets the requirements. None will be returned if there
+ * is no such message.
+ */
+ public Optional<FileRecords.TimestampAndOffset> findOffsetByTimestamp(TopicPartition tp,
+ long timestamp,
+ long startingOffset,
+ LeaderEpochFileCache leaderEpochCache) throws RemoteStorageException, IOException {
+ Uuid topicId = topicPartitionIds.get(tp);
+ if (topicId == null) {
+ throw new KafkaException("Topic id does not exist for topic partition: " + tp);
+ }
+
+ // Get the respective epoch in which the starting-offset exists.
+ OptionalInt maybeEpoch = leaderEpochCache.epochForOffset(startingOffset);
+ while (maybeEpoch.isPresent()) {
+ int epoch = maybeEpoch.getAsInt();
+
+ Iterator<RemoteLogSegmentMetadata> iterator = remoteLogMetadataManager.listRemoteLogSegments(new TopicIdPartition(topicId, tp), epoch);
+ while (iterator.hasNext()) {
+ RemoteLogSegmentMetadata rlsMetadata = iterator.next();
+ if (rlsMetadata.maxTimestampMs() >= timestamp && rlsMetadata.endOffset() >= startingOffset) {
+ return lookupTimestamp(rlsMetadata, timestamp, startingOffset);
+ }
+ }
+
+ // Move to the next epoch if not found with the current epoch.
+ maybeEpoch = leaderEpochCache.nextEpoch(epoch);
+ }
+
+ return Optional.empty();
+ }
+
+ private static abstract class CancellableRunnable implements Runnable {
+ private volatile boolean cancelled = false;
+
+ public void cancel() {
+ cancelled = true;
+ }
+
+ public boolean isCancelled() {
+ return cancelled;
+ }
+ }
+
+ /**
+ * Returns the leader epoch checkpoint by truncating with the given start[exclusive] and end[inclusive] offset
+ *
+ * @param log The actual log from where to take the leader-epoch checkpoint
+ * @param startOffset The start offset of the checkpoint file (exclusive in the truncation).
+ * If start offset is 6, then it will retain an entry at offset 6.
+ * @param endOffset The end offset of the checkpoint file (inclusive in the truncation)
+ * If end offset is 100, then it will remove the entries greater than or equal to 100.
+ * @return the truncated leader epoch checkpoint
+ */
+
+ InMemoryLeaderEpochCheckpoint getLeaderEpochCheckpoint(UnifiedLog log, long startOffset, long endOffset) {
+ InMemoryLeaderEpochCheckpoint checkpoint = new InMemoryLeaderEpochCheckpoint();
+ if (log.leaderEpochCache().isDefined()) {
+ LeaderEpochFileCache cache = log.leaderEpochCache().get().writeTo(checkpoint);
+ if (startOffset >= 0) {
+ cache.truncateFromStart(startOffset);
+ }
+ cache.truncateFromEnd(endOffset);
+ }
+
+ return checkpoint;
+ }
+
+ private class RLMTask extends CancellableRunnable {
+
+ private final TopicIdPartition topicIdPartition;
+ private final Logger logger;
+
+ private volatile int leaderEpoch = -1;
+
+ public RLMTask(TopicIdPartition topicIdPartition) {
+ this.topicIdPartition = topicIdPartition;
+ LogContext logContext = new LogContext("[RemoteLogManager=" + brokerId + " partition=" + topicIdPartition + "] ");
+ logger = logContext.logger(LeaderEpochFileCache.class);
+ }
+
+ private boolean isLeader() {
+ return leaderEpoch >= 0;
+ }
+
+ // The readOffset is None initially for a new leader RLMTask, and needs to be fetched inside the task's run() method.
+ private volatile OptionalLong copiedOffsetOption = OptionalLong.empty();
+
+ public void convertToLeader(int leaderEpochVal) {
+ if (leaderEpochVal < 0) {
+ throw new KafkaException("leaderEpoch value for topic partition $tpId can not be negative");
+ }
+ if (this.leaderEpoch != leaderEpochVal) {
+ leaderEpoch = leaderEpochVal;
+ }
+ // Reset readOffset, so that it is set in next run of RLMTask
+ copiedOffsetOption = OptionalLong.empty();
+ }
+
+ public void convertToFollower() {
+ leaderEpoch = -1;
+ }
+
+ private void maybeUpdateReadOffset() throws RemoteStorageException {
+ if (!copiedOffsetOption.isPresent()) {
+ logger.info("Find the highest remote offset for partition: {} after becoming leader, leaderEpoch: {}", topicIdPartition, leaderEpoch);
+
+ // This is found by traversing from the latest leader epoch from leader epoch history and find the highest offset
+ // of a segment with that epoch copied into remote storage. If it can not find an entry then it checks for the
+ // previous leader epoch till it finds an entry, If there are no entries till the earliest leader epoch in leader
+ // epoch cache then it starts copying the segments from the earliest epoch entry’s offset.
+ copiedOffsetOption = OptionalLong.of(findHighestRemoteOffset(topicIdPartition));
+ }
+ }
+
+ public void handleCopyLogSegmentsToRemote() throws InterruptedException {
+ if (isCancelled())
+ return;
+
+ try {
+ maybeUpdateReadOffset();
+ long copiedOffset = copiedOffsetOption.getAsLong();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (!maybeLog.isPresent()) {
+ return;
+ }
+
+ UnifiedLog log = maybeLog.get();
+
+ // LSO indicates the offset below are ready to be consumed(high-watermark or committed)
+ long lso = log.lastStableOffset();
+ if (lso < 0) {
+ logger.warn("lastStableOffset for partition {} is {}, which should not be negative.", topicIdPartition, lso);
+ } else if (lso > 0 && copiedOffset < lso) {
+ // Copy segments only till the last-stable-offset as remote storage should contain only committed/acked
+ // messages
+ long toOffset = lso;
+ logger.debug("Checking for segments to copy, copiedOffset: {} and toOffset: {}", copiedOffset, toOffset);
+ long activeSegBaseOffset = log.activeSegment().baseOffset();
+ // log-start-offset can be ahead of the read-offset, when:
+ // 1) log-start-offset gets incremented via delete-records API (or)
+ // 2) enabling the remote log for the first time
+ long fromOffset = Math.max(copiedOffset + 1, log.logStartOffset());
+ ArrayList<LogSegment> sortedSegments = new ArrayList<>(JavaConverters.asJavaCollection(log.logSegments(fromOffset, toOffset)));
+ sortedSegments.sort(Comparator.comparingLong(LogSegment::baseOffset));
+ List<Long> sortedBaseOffsets = sortedSegments.stream().map(x -> x.baseOffset()).collect(Collectors.toList());
+ int activeSegIndex = Collections.binarySearch(sortedBaseOffsets, activeSegBaseOffset);
+
+ // sortedSegments becomes empty list when fromOffset and toOffset are same, and activeSegIndex becomes -1
+ if (activeSegIndex < 0) {
+ logger.debug("No segments found to be copied for partition {} with copiedOffset: {} and active segment's base-offset: {}",
+ topicIdPartition, copiedOffset, activeSegBaseOffset);
+ } else {
+ for (LogSegment segment : sortedSegments.subList(0, activeSegIndex)) {
+ if (isCancelled() || !isLeader()) {
+ logger.info("Skipping copying log segments as the current task state is changed, cancelled: {} leader:{}",
+ isCancelled(), isLeader());
+ return;
+ }
+
+ handleCopyLogSegment(log, segment);
+ }
+ }
+ } else {
+ logger.debug("Skipping copying segments, current read-offset:{}, and LSO:{}", copiedOffset, lso);
+ }
+ } catch (InterruptedException ex) {
+ throw ex;
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.error("Error occurred while copying log segments of partition: {}", topicIdPartition, ex);
+ }
+ }
+ }
+
+ private void handleCopyLogSegment(UnifiedLog log, LogSegment segment) throws InterruptedException, ExecutionException, RemoteStorageException, IOException {
+ File logFile = segment.log().file();
+ String logFileName = logFile.getName();
+
+ logger.info("Copying {} to remote storage.", logFileName);
+ RemoteLogSegmentId id = new RemoteLogSegmentId(topicIdPartition, Uuid.randomUuid());
+
+ long nextOffset = segment.readNextOffset();
+ long endOffset = nextOffset - 1;
+ File producerIdSnapshotFile = log.producerStateManager().fetchSnapshot(nextOffset).orElse(null);
+
+ List<EpochEntry> epochEntries = getLeaderEpochCheckpoint(log, segment.baseOffset(), nextOffset).read();
+ Map<Integer, Long> segmentLeaderEpochs = new HashMap<>(epochEntries.size());
+ epochEntries.forEach(entry -> segmentLeaderEpochs.put(entry.epoch, entry.startOffset));
+
+ RemoteLogSegmentMetadata copySegmentStartedRlsm = new RemoteLogSegmentMetadata(id, segment.baseOffset(), endOffset,
+ segment.largestTimestamp(), brokerId, time.milliseconds(), segment.log().sizeInBytes(),
+ segmentLeaderEpochs);
+
+ remoteLogMetadataManager.addRemoteLogSegmentMetadata(copySegmentStartedRlsm).get();
+
+ ByteBuffer leaderEpochsIndex = getLeaderEpochCheckpoint(log, -1, nextOffset).readAsByteBuffer();
+ LogSegmentData segmentData = new LogSegmentData(logFile.toPath(), toPathIfExists(segment.lazyOffsetIndex().get().file()),
+ toPathIfExists(segment.lazyTimeIndex().get().file()), Optional.ofNullable(toPathIfExists(segment.txnIndex().file())),
+ producerIdSnapshotFile.toPath(), leaderEpochsIndex);
+ remoteLogStorageManager.copyLogSegmentData(copySegmentStartedRlsm, segmentData);
+
+ RemoteLogSegmentMetadataUpdate copySegmentFinishedRlsm = new RemoteLogSegmentMetadataUpdate(id, time.milliseconds(),
+ RemoteLogSegmentState.COPY_SEGMENT_FINISHED, brokerId);
+
+ remoteLogMetadataManager.updateRemoteLogSegmentMetadata(copySegmentFinishedRlsm).get();
+
+ copiedOffsetOption = OptionalLong.of(endOffset);
+ log.updateHighestOffsetInRemoteStorage(endOffset);
+ logger.info("Copied {} to remote storage with segment-id: {}", logFileName, copySegmentFinishedRlsm.remoteLogSegmentId());
+ }
+
+ private Path toPathIfExists(File file) {
+ return file.exists() ? file.toPath() : null;
+ }
+
+ public void run() {
+ if (isCancelled())
+ return;
+
+ try {
+ if (isLeader()) {
+ // Copy log segments to remote storage
+ handleCopyLogSegmentsToRemote();
+ }
+ } catch (InterruptedException ex) {
+ if (!isCancelled()) {
+ logger.warn("Current thread for topic-partition-id {} is interrupted, this task won't be rescheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ } catch (Exception ex) {
+ if (!isCancelled()) {
+ logger.warn("Current task for topic-partition {} received error but it will be scheduled. " +
+ "Reason: {}", topicIdPartition, ex.getMessage());
+ }
+ }
+ }
+
+ public String toString() {
+ return this.getClass().toString() + "[" + topicIdPartition + "]";
+ }
+ }
+
+ public long findHighestRemoteOffset(TopicIdPartition topicIdPartition) throws RemoteStorageException {
+ Optional<Long> offset = Optional.empty();
+ Optional<UnifiedLog> maybeLog = fetchLog.apply(topicIdPartition.topicPartition());
+ if (maybeLog.isPresent()) {
+ UnifiedLog log = maybeLog.get();
+ Option<LeaderEpochFileCache> maybeLeaderEpochFileCache = log.leaderEpochCache();
+ if (maybeLeaderEpochFileCache.isDefined()) {
+ LeaderEpochFileCache cache = maybeLeaderEpochFileCache.get();
+ OptionalInt epoch = cache.latestEpoch();
+ while (!offset.isPresent() && epoch.isPresent()) {
+ offset = remoteLogMetadataManager.highestOffsetForEpoch(topicIdPartition, epoch.getAsInt());
+ epoch = cache.previousEpoch(epoch.getAsInt());
+ }
+ }
+ }
+
+ return offset.orElse(-1L);
+ }
+
+ private void doHandleLeaderOrFollowerPartitions(TopicIdPartition topicPartition,
+ Consumer<RLMTask> convertToLeaderOrFollower) {
+ RLMTaskWithFuture rlmTaskWithFuture = leaderOrFollowerTasks.computeIfAbsent(topicPartition,
+ topicIdPartition -> {
+ RLMTask task = new RLMTask(topicIdPartition);
+ // set this upfront when it is getting initialized instead of doing it after scheduling.
+ convertToLeaderOrFollower.accept(task);
+ LOGGER.info("Created a new task: {} and getting scheduled", task);
+ ScheduledFuture future = rlmScheduledThreadPool.scheduleWithFixedDelay(task, 0, delayInMs, TimeUnit.MILLISECONDS);
+ return new RLMTaskWithFuture(task, future);
+
+ }
+ );
+ convertToLeaderOrFollower.accept(rlmTaskWithFuture.rlmTask);
+ }
+
+ private static class RLMTaskWithFuture {
+
+ private final RLMTask rlmTask;
+ private final Future<?> future;
+
+ RLMTaskWithFuture(RLMTask rlmTask, Future<?> future) {
+ this.rlmTask = rlmTask;
+ this.future = future;
+ }
+
+ public void cancel() {
+ rlmTask.cancel();
+ try {
+ future.cancel(true);
+ } catch (Exception ex) {
+ LOGGER.error("Error occurred while canceling the task: {}", rlmTask, ex);
+ }
+ }
+
+ }
+
+ private static class InMemoryLeaderEpochCheckpoint implements LeaderEpochCheckpoint {
+ private List<EpochEntry> epochs = Collections.emptyList();
+
+ public void write(Collection<EpochEntry> epochs) {
+ this.epochs = new ArrayList<>(epochs);
+ }
+
+ public List<EpochEntry> read() {
+ return Collections.unmodifiableList(epochs);
+ }
+
+ public ByteBuffer readAsByteBuffer() throws IOException {
+ ByteArrayOutputStream stream = new ByteArrayOutputStream();
+ try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(stream, StandardCharsets.UTF_8));) {
+ CheckpointFile.CheckpointWriteBuffer<EpochEntry> writeBuffer = new CheckpointFile.CheckpointWriteBuffer<>(writer, 0, LeaderEpochCheckpointFile.FORMATTER);
+ writeBuffer.write(epochs);
+ writer.flush();
+ }
+
+ return ByteBuffer.wrap(stream.toByteArray());
+ }
+ }
+
+ /**
+ * Closes and releases all the resources like RemoterStorageManager and RemoteLogMetadataManager.
+ */
+ public void close() {
+ synchronized (this) {
+ if (!closed) {
+ leaderOrFollowerTasks.values().forEach(RLMTaskWithFuture::cancel);
+ Utils.closeQuietly(remoteLogStorageManager, "RemoteLogStorageManager");
+ Utils.closeQuietly(remoteLogMetadataManager, "RemoteLogMetadataManager");
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
+ try {
+ rlmScheduledThreadPool.shutdown();
+ } catch (InterruptedException e) {
+ // ignore
+ }
+ leaderOrFollowerTasks.clear();
+ closed = true;
+ }
+ }
+ }
+
+ private static class RLMScheduledThreadPool {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(RLMScheduledThreadPool.class);
+ private final int poolSize;
+ private final ScheduledThreadPoolExecutor scheduledThreadPool;
+
+ public RLMScheduledThreadPool(int poolSize) {
+ this.poolSize = poolSize;
+ scheduledThreadPool = createPool();
+ }
+
+ private ScheduledThreadPoolExecutor createPool() {
+ ScheduledThreadPoolExecutor threadPool = new ScheduledThreadPoolExecutor(poolSize);
+ threadPool.setRemoveOnCancelPolicy(true);
+ threadPool.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
+ threadPool.setContinueExistingPeriodicTasksAfterShutdownPolicy(false);
+ threadPool.setThreadFactory(new ThreadFactory() {
+ private final AtomicInteger sequence = new AtomicInteger();
+
+ public Thread newThread(Runnable r) {
+ return KafkaThread.daemon("kafka-rlm-thread-pool-" + sequence.incrementAndGet(), r);
+ }
+ });
+
+ return threadPool;
+ }
+
+ public void resizePool(int size) {
Review Comment:
This is used for dynamic config changes that we planned to add later, removed it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org