You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/01 02:51:22 UTC
[GitHub] [pulsar] liangyepianzhou opened a new pull request, #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
liangyepianzhou opened a new pull request, #18273:
URL: https://github.com/apache/pulsar/pull/18273
Master Issue: https://github.com/apache/pulsar/issues/16913
### Motivation
Implement an abortedTxnProcessor to handle the storage of the aborted transaction ID.
### Modifications
The structure overview:
The main idea is to move the logic of the operation of checking and persistent aborted transaction IDs(take snapshots) and the operation of updating maxReadPosition into the AbortedTxnProcessor.
And the AbortedTxnProcessor can be implemented in different designs.
**Add `persistentWorker` to handle snapshot persistenting** ：
<img width="1003" alt="image" src="https://user-images.githubusercontent.com/55571188/198528131-3cde19bc-2034-4693-a8b1-4d6345e6db36.png">
The first four items below are the corresponding four tasks in the figure. The fifth item is not strictly a task, but a part of the first two tasks.
* takeSnapshotSegmentAsync -> writeSnapshotSegmentAsync
* These two method is used to persist the snapshot segment.
* deleteSnapshotSegment
* This method is used to delete the snapshot segment.
* updateIndexMetadataForTheLastSnapshot
* Using to update index metadata (the latest snapshot).
* clearSnapshotSegmentAndIndexes
* Delete all segments and then delete the index of this topic.
* updateSnapshotIndex
* Called by the deleteSnapshotSegment and writeSnapshotSegmentAsync. Do update the index after writing snapshot segment.
* Called by recovery as a compensation mechanism for updating the index.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [x] `doc-not-needed`
(Please explain why)
### Matching PR in the forked repository
PR in forked repository: https://github.com/liangyepianzhou/pulsar/pull/7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095332335
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
Review Comment:
if the size is 10, how many aborted txn in one segment? I think one segment should have at-least 2 aborted txns in the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099618974
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ //The process will check whether the snapshot segment is null,
+ // and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ }));
+ }
+ return FutureUtil.waitForAll(results);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
Review Comment:
Why will the result of the method updateSnapshotIndex never fail?
It returns the result of `indexesWriter.writeAsync(topic.getName(), snapshotIndexes);`. So there is no existing case that there are no indexes after the transaction messages are sent to the topic.
```java
return snapshotIndexWriterFuture
.thenCompose((indexesWriter) -> {
snapshotIndexes.setIndexList(indexList);
snapshotIndexes.setSnapshot(snapshotSegment);
return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
})
.thenRun(() -> lastSnapshotTimestamps = System.currentTimeMillis())
.exceptionally(e -> {
log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
return null;
});
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095346391
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
Review Comment:
This is a mistake. In this case, only one aborted txn ID in a segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#issuecomment-1367154135
>Create a delete task for each segment, and update the index within each delete task. This method costs too much.
In factly, this logic in `clearSnapshotSegmentAndIndexes` is executed in a single task.
There does not open new tasks for each segment deleting and index updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1073291903
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
Review Comment:
Can you add a note to describe how this quantity is calculated?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
Review Comment:
what that means?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
Review Comment:
```suggestion
TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
Review Comment:
```suggestion
private final int maxAbortedTxnsPerSegment;
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
Review Comment:
recover fail has already close this topic
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
Review Comment:
why not read first, read success put the persistent index into the memory index, read fail doesn't put
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
+ index.getAbortedMarkLedgerID(),
+ index.getAbortedMarkEntryID()));
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
Review Comment:
```suggestion
.thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures)).thenCompose((i) -> {
```
otherwise we can't catch the openManagedLedgerFuture exceptio
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
Review Comment:
```suggestion
public void putAbortedTxnAndPosition(TxnID txnId, PositionImpl position) {
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
Review Comment:
```suggestion
private LinkedList<TxnID> unsealedTxnIds;
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
Review Comment:
```suggestion
aborts.remove(0);
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
Review Comment:
why need this method?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
+ index.getAbortedMarkLedgerID(),
+ index.getAbortedMarkEntryID()));
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ //This is a compensation mechanism for deleting the segment
+ // but not successfully updating the index.
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ //Append the aborted txn IDs in the index metadata
+ //can keep the order of the aborted txn in the aborts.
+ //So that we can trim the expired snapshot segment in aborts
+ // according to the latest txn IDs
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ //Update index is can be canceled when the task queue is not empty, so it should be executed immediately
+ // instead of taking in queue. If the task queue is not empty, execute the task from the queue.
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ //If the operation type is clear, all the operation in the queue is meaningless.
+ case Clear -> {
+ //Do not clear the snapshots if the topic is used.
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.TransactionBufferClosedException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) return;
+ OperationType operationType = taskQueue.getFirst().getKey();
Review Comment:
still will throw the NullPointException, two thread will get the firstKey.
appendTask thread and topic.getBrokerService().getPulsar().getTransactionExecutorProvider().getExecutor(this)
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,615 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ //Cumulative the segment capital according to its size configuration.
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() >= transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
+ index.getAbortedMarkLedgerID(),
+ index.getAbortedMarkEntryID()));
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ //This is a compensation mechanism for deleting the segment
+ // but not successfully updating the index.
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ //Append the aborted txn IDs in the index metadata
+ //can keep the order of the aborted txn in the aborts.
+ //So that we can trim the expired snapshot segment in aborts
+ // according to the latest txn IDs
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ //Update index is can be canceled when the task queue is not empty, so it should be executed immediately
+ // instead of taking in queue. If the task queue is not empty, execute the task from the queue.
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ //If the operation type is clear, all the operation in the queue is meaningless.
+ case Clear -> {
+ //Do not clear the snapshots if the topic is used.
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.TransactionBufferClosedException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) return;
+ OperationType operationType = taskQueue.getFirst().getKey();
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ taskQueue.getFirst().getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), operationType.name(), throwable);
+ }
+ taskQueue.getFirst().getRight().getKey().completeExceptionally(throwable);
+ } else {
+ taskQueue.getFirst().getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ //The process will check whether the snapshot segment is null,
+ // and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ }));
+ }
+ return FutureUtil.waitForAll(results);
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ //Delete all segment
+ return persistentWorker.deleteSnapshotSegment(segmentIndex.keySet().stream().toList())
+ //Delete index
+ .thenCompose((ignore) -> snapshotIndexWriterFuture
+ .thenCompose(indexesWriter -> indexesWriter.writeAsync(topic.getName(), null)))
+ .thenRun(() ->
+ log.debug("Successes to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName()))
+ .exceptionally(e -> {
+ log.error("Failed to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName(), e);
+ return null;
+ });
+ }
+
+
+ CompletableFuture<Void> closeAsync() {
+ return CompletableFuture.allOf(
+ this.snapshotIndexWriterFuture.thenCompose(SystemTopicClient.Writer::closeAsync),
+ this.snapshotSegmentsWriterFuture.thenCompose(SystemTopicClient.Writer::closeAsync));
+ }
+ }
+
+ private LinkedList<TxnID> convertTypeToTxnID(List<TxnIDData> snapshotSegment) {
+ LinkedList<TxnID> abortedTxns = new LinkedList<>();
+ snapshotSegment.forEach(txnIDData ->
+ abortedTxns.add(new TxnID(txnIDData.getMostSigBits(), txnIDData.getLeastSigBits())));
+ return abortedTxns;
+ }
+
+ private List<TxnIDData> convertTypeToTxnIDData(LinkedList<TxnID> abortedTxns) {
Review Comment:
```suggestion
private List<TxnIDData> convertTypeToTxnIDData(List<TxnID> abortedTxns) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099611429
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
Review Comment:
There is no `LinkedSet` in the JAVA library.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099622315
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
Review Comment:
If there is another operation executing, this operation can be canceled. It is just a timer worker. Completing it is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095340496
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/AbortTxnProcessorTest.java:
##########
@@ -0,0 +1,147 @@
+package org.apache.pulsar.broker.transaction;
+
+import java.lang.reflect.Field;
+import java.util.LinkedList;
+import java.util.NavigableMap;
+import java.util.Queue;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.impl.SnapshotSegmentAbortedTxnProcessorImpl;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+@Slf4j
+public class AbortTxnProcessorTest extends TransactionTestBase {
+
+ private static final String PROCESSOR_TOPIC = "persistent://" + NAMESPACE1 + "/abortedTxnProcessor";
+ private static final int SEGMENT_SIZE = 5;
+ private PulsarService pulsarService = null;
+
+ @Override
+ @BeforeClass
+ protected void setup() throws Exception {
+ setUpBase(1, 1, PROCESSOR_TOPIC, 0);
+ this.pulsarService = getPulsarServiceList().get(0);
+ this.pulsarService.getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.pulsarService.getConfig().setTransactionBufferSnapshotSegmentSize(8 + PROCESSOR_TOPIC.length() + 5 * 3);
+ }
+
+ @Override
+ @AfterClass
+ protected void cleanup() throws Exception {
+ super.internalCleanup();
+ }
+
+ /**
+ * Test api:
+ * 1. putAbortedTxnAndPosition
+ * 2. checkAbortedTransaction
+ * 3. takeAbortedTxnsSnapshot
+ * 4. recoverFromSnapshot
+ * 5. trimExpiredAbortedTxns
+ * @throws Exception
+ */
+ @Test
+ public void testPutAbortedTxnIntoProcessor() throws Exception {
+ PersistentTopic persistentTopic = (PersistentTopic) pulsarService.getBrokerService()
+ .getTopic(PROCESSOR_TOPIC, false).get().get();
+ AbortedTxnProcessor processor = new SnapshotSegmentAbortedTxnProcessorImpl(persistentTopic);
+ //1. prepare test data.
+ //1.1 Put 10 aborted txn IDs to persistent two sealed segments.
+ for (int i = 0; i < 10; i++) {
+ TxnID txnID = new TxnID(0, i);
+ PositionImpl position = new PositionImpl(0,i);
+ processor.putAbortedTxnAndPosition(txnID, position);
+ }
+ //1.2 Put 4 aborted txn IDs into the unsealed segment.
+ for (int i = 10; i < 14; i++) {
+ TxnID txnID = new TxnID(0, i);
+ PositionImpl position = new PositionImpl(0,i);
+ processor.putAbortedTxnAndPosition(txnID, position);
+ }
+ //1.3 Verify the common data flow
+ verifyAbortedTxnIDAndSegmentIndex(processor,0,14);
+ //2. Take the latest snapshot and verify recover from snapshot
+ AbortedTxnProcessor newProcessor = new SnapshotSegmentAbortedTxnProcessorImpl(persistentTopic);
+ PositionImpl maxReadPosition = new PositionImpl(0, 14);
+ //2.1 Avoid update operation being canceled.
+ waitTaskExecuteCompletely(processor);
+ //2.2 take the latest snapshot
+ processor.takeAbortedTxnsSnapshot(maxReadPosition).get();
+ newProcessor.recoverFromSnapshot().get();
+ //Verify the recovery data flow
+ verifyAbortedTxnIDAndSegmentIndex(newProcessor,0,14);
+ //3. Delete the ledgers and then verify the date.
+ Field ledgersField = ManagedLedgerImpl.class.getDeclaredField("ledgers");
+ ledgersField.setAccessible(true);
+ NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo> ledgers =
+ (NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo>)
+ ledgersField.get(persistentTopic.getManagedLedger());
+ ledgers.forEach((k, v) -> {
Review Comment:
And there is a test deleting a ledger and verifying the snapshot segment delete in `testPutAbortedTxnIntoProcessor`.
```java
//3. Delete the ledgers and then verify the date.
Field ledgersField = ManagedLedgerImpl.class.getDeclaredField("ledgers");
ledgersField.setAccessible(true);
NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo> ledgers =
(NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo>)
ledgersField.get(persistentTopic.getManagedLedger());
ledgers.forEach((k, v) -> {
ledgers.remove(k);
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102248513
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,12 +55,17 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
this.topic = topic;
this.takeSnapshotWriter = this.topic.getBrokerService().getPulsar()
.getTransactionBufferSnapshotServiceFactory()
- .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()));
+ .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
Review Comment:
@congbobo184 Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1098374372
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
Review Comment:
In this map, key always equals value, right? if yes, why not use Set?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,651 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, (cancel) ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ //The position is already deleted.
+ if (theLatestDeletedTxnID == null) {
+ continue;
+ }
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ (cancel) -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList())));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, (cancel) -> {
+ if (!cancel) {
+ return persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
Review Comment:
If execute method `updateSnapshotIndex` fail, the CompletableFuture returned by `updateSnapshotIndexMetadata` will be a completed future, and if we call `future.get()` will return `null`, does not throw ex.
But I see these codes in the method `executeTask`:
```java
lastOperationFuture.whenComplete((ignore, throwable) -> {
if (throwable != null) {
if (log.isDebugEnabled()) {
log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
}
} else {
taskQueue.removeFirst();
}
```
the variable `throwable` in the above codes will always is `null`.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
Review Comment:
`openManagedLedgerFuture` means that all recovery tasks have been submitted, right?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
Review Comment:
should we change the variable name `snapshotIndexTopicName` -> `snapshoSegmentTopicName `?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ //The process will check whether the snapshot segment is null,
+ // and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ }));
+ }
+ return FutureUtil.waitForAll(results);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
Review Comment:
The result of the method `updateSnapshotIndex` will never fail now, even if the write fails, right?
If yes. Then there is a problem:
- enabled transaction
- take a snapshot when the topic loads (write fails)
- send some messages (some transactions are aborted)
- unload the topic before next take-snapshot-event
- do transaction buffer recover
- recover by transaction indexes (there is no data)
- since there are no transaction indexes, skip the recover by original topic log task
- an error occurs: This topic has canceled transactions, but there are no data in the `TB.aborts`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
Review Comment:
The operation of updating indexes has yet to be complete when this future is complete, right?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
Review Comment:
The operation of updating indexes has yet to be complete when this future is complete, right?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
Review Comment:
Maybe we can save once `convertTypeToTxnID` by variable `unsealedTxnIds`?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
Review Comment:
two questions:
- when this line is executed, the state of the topic will already be closed or fenced, right?
- All type of adding task is in the lock `synchronized (TopicTransactionBuffer.this)` except clear, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095343870
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.TransactionBufferClosedException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) return;
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ if (firstTask == null) return;
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the task queue.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
Review Comment:
Notes adding in `writeSnapshotSegmentAsync`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095957896
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -550,7 +558,7 @@ private void checkSnapshotCount(TopicName topicName, boolean hasSnapshot,
CompletableFuture<Long> compactionFuture = (CompletableFuture<Long>) field.get(persistentTopic);
Awaitility.await().untilAsserted(() -> assertTrue(compactionFuture.isDone()));
- Reader<TransactionBufferSnapshot> reader = pulsarClient.newReader(Schema.AVRO(TransactionBufferSnapshot.class))
+ Reader<GenericRecord> reader = pulsarClient.newReader(Schema.AUTO_CONSUME())
Review Comment:
Add the test logic in `SegmentAbortedTxnProcessorTest::testClearSnapshotSegments`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060250181
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
Review Comment:
if we don't have any PositionImpl to trim, we don't need to create that list
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
Review Comment:
in updateSnapshotIndex, snapshotIndexWriterFuture is an async created, the index will be updated at this time when snapshotIndexWriterFuture is creating. and use snapshotIndexWriterFuture the write may write out of order.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
Review Comment:
if the list is empty, we don't need to appendTask
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
Review Comment:
why this type is close not clear
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
Review Comment:
entry should release, otherwise will OOM
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
Review Comment:
why can't return `CompletableFuture<Void>` directly
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
Review Comment:
the positionNeedToDeletes is abortMarkPersistentPostion not maxReadPosition how we can get the correct segement sequenceId
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
Review Comment:
if create fail, we should close this topic
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
Review Comment:
if the position is invalid, the read op will throw exception or return null?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
Review Comment:
now we have two thread to use this method, appendTask() and timer, if one thread change the state to WritingSegment, and the second thread also can change state to WritingSegment when the task is finished. and the first thread removes the first task and change the state to `NONE`, the second task is DeletingSegment, how we change the state to `NONE`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060398088
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
Review Comment:
We do not need to wait for the result of the operation, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060675874
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
Review Comment:
`snapshotIndexWriterFuture` is only called at the single thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1021044374
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
Review Comment:
Maybe it is better to call this method `delete xxx`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1059216957
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
Review Comment:
No, it throws an `NonRecoverableLedgerException` exception if the entry does not exist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1052881061
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,8 +56,8 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
}
@Override
- public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl position) {
- aborts.put(abortedTxnId, position);
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ aborts.put(abortedTxnId, this.topic.getMaxReadPosition());
Review Comment:
This is the original single snapshot aborted processor which stores the aborted txns and maxReadPosition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#issuecomment-1446270913
This PR introduced a new flaky test #19650, please fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099861973
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
Review Comment:
Doesn't the index need to be updated after deleting the segments?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ //The process will check whether the snapshot segment is null,
+ // and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ }));
+ }
+ return FutureUtil.waitForAll(results);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
Review Comment:
the arg `List<TransactionBufferSnapshotIndex> indexList` of method `updateSnapshotIndex` is unnecessary, right?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
Review Comment:
##### Additional question
The operation `update indexes` will do three things:
- update indexes
- update max read position
- update unsealed aborts
But the operation `update indexes` followed by `take new segment` only do two things:
- update indexes
- update max read position
So is it okay to skip task `update unsealed aborts`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099615573
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
Review Comment:
Sorry, I can not understand this.
`UnsealedTxnIds` is not recovered in the processor. There only recover aborted transaction IDs in snapshot segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1021388094
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
Review Comment:
It can be understood to only change in `updateSnapshotIndex`.
When the latest max read position and aborted txn id are persistent in the index snapshot, lastSnapshotTimestamps will be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1021381839
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ //Delete all segment
+ while (!abortTxnSegments.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ completableFutures.add(persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
Review Comment:
Good point out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058812123
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
Review Comment:
Good idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099620734
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
Review Comment:
When this line is executed, it means the state failed to change to closed.
This close operation will be canceled, so the task in the queue should not be clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060389565
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
Review Comment:
At the beginning of the design, it was considered that `write/delete snapshots` and `update index` were non-atomic operations. The update index operation after the `write snapshot` may fail. Here re-update index is the pudding for `update index` failure after `delete snapshots`. This is an expected behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060368058
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
Review Comment:
Do you mean to add the `batch delete ` method to the `systemTopic::Writer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060680342
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1064401552
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,630 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> {
+ persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
Review Comment:
1. In the method, `appendTask` newly added Update Index tasks are ignored if the queue is not empty, but I found that multiple Update Index tasks are not interchangeable because their meta is different.
2. I wonder if there will be such a problem: the `metadata` and `indexList` attributes of Snapshot Indexes don't match, for example:
```
add task 1 {meta=1, indexList=[1]}
add task 2 {meta=2, indexList=[1,2]}
when the task 1 executed, it use attributes is {meta=1, indexList=[1,2]}
```
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,630 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> {
+ persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
+ index.getAbortedMarkLedgerID(),
+ index.getAbortedMarkEntryID()));
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });;
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ //Update index is can be canceled when the task queue is not empty, so it should be executed immediately
+ // instead of taking in queue. If the task queue is not empty, execute the task from the queue.
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ //If the operation type is clear, all the operation in the queue is meaningless.
+ case Clear -> {
+ STATE_UPDATER.set(this, OperationState.Operating);
+ taskQueue.clear();
Review Comment:
Should we complete the CompletableFuture in the queue before we clear this queue?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,630 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> {
+ persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
+ index.getAbortedMarkLedgerID(),
+ index.getAbortedMarkEntryID()));
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });;
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ //Update index is can be canceled when the task queue is not empty, so it should be executed immediately
+ // instead of taking in queue. If the task queue is not empty, execute the task from the queue.
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
Review Comment:
This task in parameters will not be executed, right? If yes, then the CompletableFuture definition in method `CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition)` will never be completed, right? We should add a test for this case: all the CompletableFuture will be completed at last.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095341292
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.TransactionBufferClosedException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) return;
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ if (firstTask == null) return;
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the task queue.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
Review Comment:
The segment will be overwritten when the processor writes the snapshot segment next time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#issuecomment-1963413109
the implementation contains multiple thread-safety issues, I filed #22116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1064566478
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,630 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> {
+ persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
Review Comment:
Good question!
The `metadata` does not need to match the `indexList`. The `metadata` contains the maxReadPosition and a list of the unsealed aborted txn ID segment. And the `indexList` stores the index of the sealed snapshot segment which writes after the writing of the snapshot segment.
So their matching relationship is:
* unsealed snapshot segment ----> maxReadPosition. (Both contains in `metadata`)
* sealed snapshot segment ----> indexList
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060389565
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
Review Comment:
At the beginning of the design, it was considered that `write/delete snapshots` and `update index` were non-atomic operations. The update index operation after the `write snapshot` may fail. Here re-update index is the pudding for `update index` failure after `delete snapshots`. This is an expected behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1092862754
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
Review Comment:
if the exceptin instanceof NonRecoverableLedgerException represent the index is Invalid right?
```suggestion
if (!exception instanceof ManagedLedgerException
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095339686
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/AbortTxnProcessorTest.java:
##########
@@ -0,0 +1,147 @@
+package org.apache.pulsar.broker.transaction;
+
+import java.lang.reflect.Field;
+import java.util.LinkedList;
+import java.util.NavigableMap;
+import java.util.Queue;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.impl.SnapshotSegmentAbortedTxnProcessorImpl;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+@Slf4j
+public class AbortTxnProcessorTest extends TransactionTestBase {
+
+ private static final String PROCESSOR_TOPIC = "persistent://" + NAMESPACE1 + "/abortedTxnProcessor";
+ private static final int SEGMENT_SIZE = 5;
+ private PulsarService pulsarService = null;
+
+ @Override
+ @BeforeClass
+ protected void setup() throws Exception {
+ setUpBase(1, 1, PROCESSOR_TOPIC, 0);
+ this.pulsarService = getPulsarServiceList().get(0);
+ this.pulsarService.getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.pulsarService.getConfig().setTransactionBufferSnapshotSegmentSize(8 + PROCESSOR_TOPIC.length() + 5 * 3);
+ }
+
+ @Override
+ @AfterClass
+ protected void cleanup() throws Exception {
+ super.internalCleanup();
+ }
+
+ /**
+ * Test api:
+ * 1. putAbortedTxnAndPosition
+ * 2. checkAbortedTransaction
+ * 3. takeAbortedTxnsSnapshot
+ * 4. recoverFromSnapshot
+ * 5. trimExpiredAbortedTxns
+ * @throws Exception
+ */
+ @Test
+ public void testPutAbortedTxnIntoProcessor() throws Exception {
+ PersistentTopic persistentTopic = (PersistentTopic) pulsarService.getBrokerService()
+ .getTopic(PROCESSOR_TOPIC, false).get().get();
+ AbortedTxnProcessor processor = new SnapshotSegmentAbortedTxnProcessorImpl(persistentTopic);
+ //1. prepare test data.
+ //1.1 Put 10 aborted txn IDs to persistent two sealed segments.
+ for (int i = 0; i < 10; i++) {
+ TxnID txnID = new TxnID(0, i);
+ PositionImpl position = new PositionImpl(0,i);
+ processor.putAbortedTxnAndPosition(txnID, position);
+ }
+ //1.2 Put 4 aborted txn IDs into the unsealed segment.
+ for (int i = 10; i < 14; i++) {
+ TxnID txnID = new TxnID(0, i);
+ PositionImpl position = new PositionImpl(0,i);
+ processor.putAbortedTxnAndPosition(txnID, position);
+ }
+ //1.3 Verify the common data flow
+ verifyAbortedTxnIDAndSegmentIndex(processor,0,14);
+ //2. Take the latest snapshot and verify recover from snapshot
+ AbortedTxnProcessor newProcessor = new SnapshotSegmentAbortedTxnProcessorImpl(persistentTopic);
+ PositionImpl maxReadPosition = new PositionImpl(0, 14);
+ //2.1 Avoid update operation being canceled.
+ waitTaskExecuteCompletely(processor);
+ //2.2 take the latest snapshot
+ processor.takeAbortedTxnsSnapshot(maxReadPosition).get();
+ newProcessor.recoverFromSnapshot().get();
+ //Verify the recovery data flow
+ verifyAbortedTxnIDAndSegmentIndex(newProcessor,0,14);
+ //3. Delete the ledgers and then verify the date.
+ Field ledgersField = ManagedLedgerImpl.class.getDeclaredField("ledgers");
+ ledgersField.setAccessible(true);
+ NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo> ledgers =
+ (NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo>)
+ ledgersField.get(persistentTopic.getManagedLedger());
+ ledgers.forEach((k, v) -> {
Review Comment:
There is a clear snapshot test `clearTransactionBufferSnapshotTest` in `TopicTransactionBufferRecoverTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095362131
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
+ int theCountOfSnapshotMaxTxnCount = 3;
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSnapshotSegmentSize(theSizeOfSegment);
+ this.getPulsarServiceList().get(0).getConfig()
+ .setTransactionBufferSnapshotMaxTransactionCount(theCountOfSnapshotMaxTxnCount);
+ // 1. Build producer and consumer
+ Producer<Integer> producer = pulsarClient.newProducer(Schema.INT32)
+ .topic(topic)
+ .enableBatching(false)
+ .create();
+
+ Consumer<Integer> consumer = pulsarClient.newConsumer(Schema.INT32)
+ .topic(topic)
+ .subscriptionName(subName)
+ .subscriptionType(SubscriptionType.Exclusive)
+ .subscribe();
+
+ // 2. Check the AbortedTxnProcessor workflow 10 times
+ int messageSize = theSizeOfSegment * 4;
+ for (int i = 0; i < 10; i++) {
+ MessageId maxReadMessage = null;
+ int abortedTxnSize = 0;
+ for (int j = 0; j < messageSize; j++) {
+ Transaction transaction = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.MINUTES).build().get();
+ //Half common message and half transaction message.
+ if (j % 2 == 0) {
+ MessageId messageId = producer.newMessage(transaction).value(i * 10 + j).send();
+ //And the transaction message have a half which are aborted.
+ if (RandomUtils.nextInt() % 2 == 0) {
+ transaction.abort().get();
+ abortedTxns.add(messageId);
+ abortedTxnSize++;
+ } else {
+ ongoingTxns.put(transaction, messageId);
+ if (maxReadMessage == null) {
+ //The except number of the messages that can be read
+ maxReadMessage = messageId;
+ }
+ }
+ } else {
+ producer.newMessage().value(i * 10 + j).send();
+ transaction.commit().get();
+ }
+ }
+ // 2.1 Receive all message before the maxReadPosition to verify the correctness of the max read position.
+ int hasReceived = 0;
+ while (true) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ if (message != null) {
+ Assert.assertTrue(message.getMessageId().compareTo(maxReadMessage) < 0);
+ hasReceived ++;
+ } else {
+ break;
+ }
+ }
+ //2.2 Commit all ongoing transaction and verify that the consumer can receive all rest message
+ // expect for aborted txn message.
+ for (Transaction ongoingTxn: ongoingTxns.keySet()) {
+ ongoingTxn.commit().get();
+ }
+ ongoingTxns.clear();
+ for (int k = hasReceived; k < messageSize - abortedTxnSize; k++) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ assertNotNull(message);
+ assertFalse(abortedTxns.contains(message.getMessageId()));
+ }
+ }
+ // 3. After the topic unload, the consumer can receive all the messages in the 10 tests
+ // expect for the aborted transaction messages.
Review Comment:
> before unload, check the aborted txn, segment, index stats from the processor
The processor's internal stat verity can be found in `SegmentAbortedTxnProcessorTest`.
This test will verify the processor can run properly end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1101325094
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,12 +55,17 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
this.topic = topic;
this.takeSnapshotWriter = this.topic.getBrokerService().getPulsar()
.getTransactionBufferSnapshotServiceFactory()
- .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()));
+ .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
Review Comment:
If the snapshot writer created fails, the processor or the transaction buffer is unusable.
So we should catch the exception and close the topic here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1101334210
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
Fixed, I make it only update indexes when it is the latest task in the task queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102267957
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
I have thought about this. So I first set maxReadPosition and unsealed Aborts as local variables, and then make a judgment.
```
PositionImpl maxReadPosition = topic.getMaxReadPosition();
List<TxnIDData> aborts = convertTypeToTxnIDData(unsealedTxnIds);
if (taskQueue.size() != 1) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099632191
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ //The process will check whether the snapshot segment is null,
+ // and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ }));
+ }
+ return FutureUtil.waitForAll(results);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
Review Comment:
> Why will the result of the method updateSnapshotIndex never fail?
you can run this test:
```java
@Test
public void testFuture() throws Exception{
CompletableFuture<Integer> future = CompletableFuture.failedFuture(new NullPointerException("null"));
CompletableFuture<Integer> future2 = future.exceptionally((ex) -> {
return 2;
});
AtomicBoolean exTwice = new AtomicBoolean();
future2.exceptionally(ex -> {
exTwice.set(true);
return 2;
}).get();
assertTrue(exTwice.get());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058492478
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
Review Comment:
When deleting a Segment, we delete the Segment first and then update the Index. If the delete Segment succeeds, but the updated Index fails, then the error occurs here. We can determine that entry is invalid by this way: `TransactionBufferSnapshotIndex.maxReadPositionLedgerID < topic.ledgers.minLedgerId`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060679211
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
Review Comment:
We need is the result of the task rather than the operation of `appendTask` so there can not return CompletableFuture<Void> directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060410898
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
Review Comment:
Good catch! I think we should change the key of the indexes to `abortMarkPersistentPostion`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095326157
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/AbortTxnProcessorTest.java:
##########
@@ -0,0 +1,147 @@
+package org.apache.pulsar.broker.transaction;
+
+import java.lang.reflect.Field;
+import java.util.LinkedList;
+import java.util.NavigableMap;
+import java.util.Queue;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.impl.SnapshotSegmentAbortedTxnProcessorImpl;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+@Slf4j
+public class AbortTxnProcessorTest extends TransactionTestBase {
+
+ private static final String PROCESSOR_TOPIC = "persistent://" + NAMESPACE1 + "/abortedTxnProcessor";
+ private static final int SEGMENT_SIZE = 5;
+ private PulsarService pulsarService = null;
+
+ @Override
+ @BeforeClass
+ protected void setup() throws Exception {
+ setUpBase(1, 1, PROCESSOR_TOPIC, 0);
+ this.pulsarService = getPulsarServiceList().get(0);
+ this.pulsarService.getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.pulsarService.getConfig().setTransactionBufferSnapshotSegmentSize(8 + PROCESSOR_TOPIC.length() + 5 * 3);
+ }
+
+ @Override
+ @AfterClass
+ protected void cleanup() throws Exception {
+ super.internalCleanup();
+ }
+
+ /**
+ * Test api:
+ * 1. putAbortedTxnAndPosition
+ * 2. checkAbortedTransaction
+ * 3. takeAbortedTxnsSnapshot
+ * 4. recoverFromSnapshot
+ * 5. trimExpiredAbortedTxns
+ * @throws Exception
+ */
+ @Test
+ public void testPutAbortedTxnIntoProcessor() throws Exception {
+ PersistentTopic persistentTopic = (PersistentTopic) pulsarService.getBrokerService()
+ .getTopic(PROCESSOR_TOPIC, false).get().get();
+ AbortedTxnProcessor processor = new SnapshotSegmentAbortedTxnProcessorImpl(persistentTopic);
+ //1. prepare test data.
+ //1.1 Put 10 aborted txn IDs to persistent two sealed segments.
+ for (int i = 0; i < 10; i++) {
+ TxnID txnID = new TxnID(0, i);
+ PositionImpl position = new PositionImpl(0,i);
+ processor.putAbortedTxnAndPosition(txnID, position);
+ }
+ //1.2 Put 4 aborted txn IDs into the unsealed segment.
+ for (int i = 10; i < 14; i++) {
+ TxnID txnID = new TxnID(0, i);
+ PositionImpl position = new PositionImpl(0,i);
+ processor.putAbortedTxnAndPosition(txnID, position);
+ }
+ //1.3 Verify the common data flow
+ verifyAbortedTxnIDAndSegmentIndex(processor,0,14);
+ //2. Take the latest snapshot and verify recover from snapshot
+ AbortedTxnProcessor newProcessor = new SnapshotSegmentAbortedTxnProcessorImpl(persistentTopic);
+ PositionImpl maxReadPosition = new PositionImpl(0, 14);
+ //2.1 Avoid update operation being canceled.
+ waitTaskExecuteCompletely(processor);
+ //2.2 take the latest snapshot
+ processor.takeAbortedTxnsSnapshot(maxReadPosition).get();
+ newProcessor.recoverFromSnapshot().get();
+ //Verify the recovery data flow
+ verifyAbortedTxnIDAndSegmentIndex(newProcessor,0,14);
+ //3. Delete the ledgers and then verify the date.
+ Field ledgersField = ManagedLedgerImpl.class.getDeclaredField("ledgers");
+ ledgersField.setAccessible(true);
+ NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo> ledgers =
+ (NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo>)
+ ledgersField.get(persistentTopic.getManagedLedger());
+ ledgers.forEach((k, v) -> {
Review Comment:
could we verify one ledger delete, delete one snapshot segment? and add clear snapshot test.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
Review Comment:
I have fogoted, why we need executor to excute this task? please add a note for it
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.TransactionBufferClosedException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) return;
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ if (firstTask == null) return;
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the task queue.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
Review Comment:
If one segment has been written to the segment topic but the segment has not been written into the index and updated success, how can clear this segment?
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/AbortTxnProcessorTest.java:
##########
@@ -0,0 +1,147 @@
+package org.apache.pulsar.broker.transaction;
+
+import java.lang.reflect.Field;
+import java.util.LinkedList;
+import java.util.NavigableMap;
+import java.util.Queue;
+import java.util.concurrent.TimeUnit;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.proto.MLDataFormats;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.pulsar.broker.PulsarService;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.impl.SnapshotSegmentAbortedTxnProcessorImpl;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.testcontainers.shaded.org.awaitility.Awaitility;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+@Slf4j
+public class AbortTxnProcessorTest extends TransactionTestBase {
Review Comment:
```suggestion
public class SegementAbortedTxnProcessorTest extends TransactionTestBase {
```
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
+ int theCountOfSnapshotMaxTxnCount = 3;
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSnapshotSegmentSize(theSizeOfSegment);
+ this.getPulsarServiceList().get(0).getConfig()
+ .setTransactionBufferSnapshotMaxTransactionCount(theCountOfSnapshotMaxTxnCount);
+ // 1. Build producer and consumer
+ Producer<Integer> producer = pulsarClient.newProducer(Schema.INT32)
+ .topic(topic)
+ .enableBatching(false)
+ .create();
+
+ Consumer<Integer> consumer = pulsarClient.newConsumer(Schema.INT32)
+ .topic(topic)
+ .subscriptionName(subName)
+ .subscriptionType(SubscriptionType.Exclusive)
+ .subscribe();
+
+ // 2. Check the AbortedTxnProcessor workflow 10 times
+ int messageSize = theSizeOfSegment * 4;
+ for (int i = 0; i < 10; i++) {
+ MessageId maxReadMessage = null;
+ int abortedTxnSize = 0;
+ for (int j = 0; j < messageSize; j++) {
+ Transaction transaction = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.MINUTES).build().get();
+ //Half common message and half transaction message.
+ if (j % 2 == 0) {
+ MessageId messageId = producer.newMessage(transaction).value(i * 10 + j).send();
+ //And the transaction message have a half which are aborted.
+ if (RandomUtils.nextInt() % 2 == 0) {
+ transaction.abort().get();
+ abortedTxns.add(messageId);
+ abortedTxnSize++;
+ } else {
+ ongoingTxns.put(transaction, messageId);
+ if (maxReadMessage == null) {
+ //The except number of the messages that can be read
+ maxReadMessage = messageId;
+ }
+ }
+ } else {
+ producer.newMessage().value(i * 10 + j).send();
+ transaction.commit().get();
+ }
+ }
+ // 2.1 Receive all message before the maxReadPosition to verify the correctness of the max read position.
+ int hasReceived = 0;
+ while (true) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ if (message != null) {
+ Assert.assertTrue(message.getMessageId().compareTo(maxReadMessage) < 0);
+ hasReceived ++;
+ } else {
+ break;
+ }
+ }
+ //2.2 Commit all ongoing transaction and verify that the consumer can receive all rest message
+ // expect for aborted txn message.
+ for (Transaction ongoingTxn: ongoingTxns.keySet()) {
+ ongoingTxn.commit().get();
+ }
+ ongoingTxns.clear();
+ for (int k = hasReceived; k < messageSize - abortedTxnSize; k++) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ assertNotNull(message);
+ assertFalse(abortedTxns.contains(message.getMessageId()));
+ }
+ }
+ // 3. After the topic unload, the consumer can receive all the messages in the 10 tests
+ // expect for the aborted transaction messages.
Review Comment:
before unload, check the aborted txn, segment, index stats from the processor, after unload check again. and add the tirm and clear test.
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
Review Comment:
if the size is 10, how many aborted txn in one segment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099621420
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
Review Comment:
The operation of updating indexes is meaningless when there are other operations in the queue. So just completing it is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102311939
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/metadata/v2/TransactionBufferSnapshotIndex.java:
##########
@@ -29,8 +29,8 @@
@NoArgsConstructor
public class TransactionBufferSnapshotIndex {
public long sequenceID;
- public long maxReadPositionLedgerID;
- public long maxReadPositionEntryID;
- public long persistentPositionLedgerID;
- public long persistentPositionEntryID;
+ public long abortedMarkLedgerID;
+ public long abortedMarkEntryID;
+ public long segmentLedgerID;
+ public long segmentEntryID;
Review Comment:
```suggestion
public long abortedMarkLedgerID;
public long abortedMarkEntryID;
/** location(ledger id of position) of a segment data in the system topic __transaction_buffer_snapshot_segments **/
public long segmentLedgerID;
/** location(entry id of position) of a segment data in the system topic __transaction_buffer_snapshot_segments **/
public long segmentEntryID;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060368058
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
Review Comment:
Do you mean to add the `batch delete ` method to the `systemTopic::Writer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1021389862
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ //Delete all segment
+ while (!abortTxnSegments.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ completableFutures.add(persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ //Delete index
+ return FutureUtil.waitForAll(completableFutures)
+ .thenCompose((ignore) -> snapshotIndexWriterFuture
+ .thenCompose(indexesWriter -> indexesWriter.writeAsync(topic.getName(), null)))
+ .thenRun(() -> {
+ log.info("Successes to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName());
+
+ })
+ .exceptionally(e -> {
+ log.error("Failed to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName(), e);
+
+ return null;
+ });
+ }
+
+
+ CompletableFuture<Void> closeAsync() {
Review Comment:
The timer is shared by all topics in a. broker.
`this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#issuecomment-1352460298
The pr had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058800984
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
Review Comment:
If the deleted segment succeeds, but the updated index fails. `invalidIndex` will be larger than 0, and then TB will update the index again when TB recovers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058495746
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
Review Comment:
Can we do batch trim?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou closed pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou closed pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
URL: https://github.com/apache/pulsar/pull/18273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095362131
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
+ int theCountOfSnapshotMaxTxnCount = 3;
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSnapshotSegmentSize(theSizeOfSegment);
+ this.getPulsarServiceList().get(0).getConfig()
+ .setTransactionBufferSnapshotMaxTransactionCount(theCountOfSnapshotMaxTxnCount);
+ // 1. Build producer and consumer
+ Producer<Integer> producer = pulsarClient.newProducer(Schema.INT32)
+ .topic(topic)
+ .enableBatching(false)
+ .create();
+
+ Consumer<Integer> consumer = pulsarClient.newConsumer(Schema.INT32)
+ .topic(topic)
+ .subscriptionName(subName)
+ .subscriptionType(SubscriptionType.Exclusive)
+ .subscribe();
+
+ // 2. Check the AbortedTxnProcessor workflow 10 times
+ int messageSize = theSizeOfSegment * 4;
+ for (int i = 0; i < 10; i++) {
+ MessageId maxReadMessage = null;
+ int abortedTxnSize = 0;
+ for (int j = 0; j < messageSize; j++) {
+ Transaction transaction = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.MINUTES).build().get();
+ //Half common message and half transaction message.
+ if (j % 2 == 0) {
+ MessageId messageId = producer.newMessage(transaction).value(i * 10 + j).send();
+ //And the transaction message have a half which are aborted.
+ if (RandomUtils.nextInt() % 2 == 0) {
+ transaction.abort().get();
+ abortedTxns.add(messageId);
+ abortedTxnSize++;
+ } else {
+ ongoingTxns.put(transaction, messageId);
+ if (maxReadMessage == null) {
+ //The except number of the messages that can be read
+ maxReadMessage = messageId;
+ }
+ }
+ } else {
+ producer.newMessage().value(i * 10 + j).send();
+ transaction.commit().get();
+ }
+ }
+ // 2.1 Receive all message before the maxReadPosition to verify the correctness of the max read position.
+ int hasReceived = 0;
+ while (true) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ if (message != null) {
+ Assert.assertTrue(message.getMessageId().compareTo(maxReadMessage) < 0);
+ hasReceived ++;
+ } else {
+ break;
+ }
+ }
+ //2.2 Commit all ongoing transaction and verify that the consumer can receive all rest message
+ // expect for aborted txn message.
+ for (Transaction ongoingTxn: ongoingTxns.keySet()) {
+ ongoingTxn.commit().get();
+ }
+ ongoingTxns.clear();
+ for (int k = hasReceived; k < messageSize - abortedTxnSize; k++) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ assertNotNull(message);
+ assertFalse(abortedTxns.contains(message.getMessageId()));
+ }
+ }
+ // 3. After the topic unload, the consumer can receive all the messages in the 10 tests
+ // expect for the aborted transaction messages.
Review Comment:
> before unload, check the aborted txn, segment, index stats from the processor
The processor's internal stat verity can be found in `SegmentAbortedTxnProcessorTest`.
This test will verify the processor can run properly end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#issuecomment-1416338557
# [Codecov](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#18273](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b498a2d) into [master](https://codecov.io/gh/apache/pulsar/commit/492a9c3e44bef2334a77164afc8b033cc8f8d82f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (492a9c3) will **decrease** coverage by `23.56%`.
> The diff coverage is `5.23%`.
[](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #18273 +/- ##
=============================================
- Coverage 47.46% 23.90% -23.56%
+ Complexity 10727 294 -10433
=============================================
Files 711 1654 +943
Lines 69456 126369 +56913
Branches 7452 13771 +6319
=============================================
- Hits 32964 30214 -2750
- Misses 32810 91699 +58889
- Partials 3682 4456 +774
```
| Flag | Coverage Δ | |
|---|---|---|
| inttests | `23.90% <5.23%> (?)` | |
| unittests | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pulsar/broker/admin/impl/SchemasResourceBase.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL1NjaGVtYXNSZXNvdXJjZUJhc2UuamF2YQ==) | `38.23% <0.00%> (-51.67%)` | :arrow_down: |
| [...r/impl/SnapshotSegmentAbortedTxnProcessorImpl.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9TbmFwc2hvdFNlZ21lbnRBYm9ydGVkVHhuUHJvY2Vzc29ySW1wbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...er/metadata/v2/TransactionBufferSnapshotIndex.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvbWV0YWRhdGEvdjIvVHJhbnNhY3Rpb25CdWZmZXJTbmFwc2hvdEluZGV4LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...a/v2/TransactionBufferSnapshotIndexesMetadata.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvbWV0YWRhdGEvdjIvVHJhbnNhY3Rpb25CdWZmZXJTbmFwc2hvdEluZGV4ZXNNZXRhZGF0YS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../metadata/v2/TransactionBufferSnapshotSegment.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvbWV0YWRhdGEvdjIvVHJhbnNhY3Rpb25CdWZmZXJTbmFwc2hvdFNlZ21lbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...er/impl/SingleSnapshotAbortedTxnProcessorImpl.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9TaW5nbGVTbmFwc2hvdEFib3J0ZWRUeG5Qcm9jZXNzb3JJbXBsLmphdmE=) | `40.42% <28.57%> (-30.23%)` | :arrow_down: |
| [...ransaction/buffer/impl/TopicTransactionBuffer.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci90cmFuc2FjdGlvbi9idWZmZXIvaW1wbC9Ub3BpY1RyYW5zYWN0aW9uQnVmZmVyLmphdmE=) | `32.94% <28.57%> (-25.20%)` | :arrow_down: |
| [...va/org/apache/pulsar/client/impl/ConsumerImpl.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL0NvbnN1bWVySW1wbC5qYXZh) | `34.46% <90.00%> (+19.36%)` | :arrow_up: |
| [...r/service/SystemTopicTxnBufferSnapshotService.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1N5c3RlbVRvcGljVHhuQnVmZmVyU25hcHNob3RTZXJ2aWNlLmphdmE=) | `60.86% <100.00%> (-17.40%)` | :arrow_down: |
| [...er/systopic/NamespaceEventsSystemTopicFactory.java](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zeXN0b3BpYy9OYW1lc3BhY2VFdmVudHNTeXN0ZW1Ub3BpY0ZhY3RvcnkuamF2YQ==) | `71.42% <100.00%> (-1.91%)` | :arrow_down: |
| ... and [1433 more](https://codecov.io/gh/apache/pulsar/pull/18273?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099608202
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
Review Comment:
Yeap, you are right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099615573
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
Review Comment:
Sorry, I can not understand this.
`UnsealedTxnIds` is not recovered in the processor. There only recover aborted transaction IDs in snapshot segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102288139
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
| time | thread-`TB.abortTxn` | thread-`execute task which in the queue` |
| --- | --- | --- |
| 1 | | write `segment-1` |
| 2 | update `maxReadPosition` of the topic | |
| 3 | | read `maxReadPosition` of the topic |
| 4 | | read `unsealedAborts` |
| 5 | put aborts to `unsealedAborts` | |
| 6 | | update indexes |
if the broker crash after step 6: after reloading this topic, an error occurs, the `maxReadPosition` is larger than `unsealedAborts`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102299362
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
Good catch! We will lose an aborted txn ID in this case. So change the order of putting `aborted txn ID` and updating `maxReadPosition` to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102299362
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
Good catch! Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102248513
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,12 +55,17 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
this.topic = topic;
this.takeSnapshotWriter = this.topic.getBrokerService().getPulsar()
.getTransactionBufferSnapshotServiceFactory()
- .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()));
+ .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
Review Comment:
@congbobo184 Could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058918025
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,614 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
Review Comment:
why not use `ArrayList`, `LinkedList` add slowly than `ArrayList`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,614 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
Review Comment:
why is `LinkedList`, why not use `LinkedHashSet`, `LinkedList` search very slowly
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,614 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
Review Comment:
why not combime aborts use `LinkedHashMap`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1021026427
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
Review Comment:
Why not use `ceilingEntry` here?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ //Delete all segment
+ while (!abortTxnSegments.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ completableFutures.add(persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ //Delete index
+ return FutureUtil.waitForAll(completableFutures)
+ .thenCompose((ignore) -> snapshotIndexWriterFuture
+ .thenCompose(indexesWriter -> indexesWriter.writeAsync(topic.getName(), null)))
+ .thenRun(() -> {
+ log.info("Successes to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName());
+
+ })
+ .exceptionally(e -> {
+ log.error("Failed to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName(), e);
+
+ return null;
+ });
+ }
+
+
+ CompletableFuture<Void> closeAsync() {
+ return CompletableFuture.allOf(
+ this.snapshotIndexWriterFuture.thenCompose(SystemTopicClient.Writer::closeAsync),
+ this.snapshotSegmentsWriterFuture.thenCompose(SystemTopicClient.Writer::closeAsync));
+ }
+ }
+
+ private ConcurrentOpenHashSet<TxnID> deserializationFotSnapshotSegment(List<TxnIDData> snapshotSegment) {
Review Comment:
Suggestion change this method name to `convertTypeToTxnID`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ //Delete all segment
+ while (!abortTxnSegments.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ completableFutures.add(persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ //Delete index
+ return FutureUtil.waitForAll(completableFutures)
+ .thenCompose((ignore) -> snapshotIndexWriterFuture
+ .thenCompose(indexesWriter -> indexesWriter.writeAsync(topic.getName(), null)))
+ .thenRun(() -> {
+ log.info("Successes to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName());
+
+ })
+ .exceptionally(e -> {
+ log.error("Failed to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName(), e);
+
+ return null;
+ });
+ }
+
+
+ CompletableFuture<Void> closeAsync() {
+ return CompletableFuture.allOf(
+ this.snapshotIndexWriterFuture.thenCompose(SystemTopicClient.Writer::closeAsync),
+ this.snapshotSegmentsWriterFuture.thenCompose(SystemTopicClient.Writer::closeAsync));
+ }
+ }
+
+ private ConcurrentOpenHashSet<TxnID> deserializationFotSnapshotSegment(List<TxnIDData> snapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedTxns = new ConcurrentOpenHashSet<>();
+ snapshotSegment.forEach(txnIDData -> {
+ abortedTxns.add(new TxnID(txnIDData.getMostSigBits(), txnIDData.getLeastSigBits()));
+ });
+ return abortedTxns;
+ }
+
+ private List<TxnIDData> serializationForSegment(ConcurrentOpenHashSet<TxnID> abortedTxns) {
Review Comment:
Suggestion changing this method name to `convertTypeToTxnIDData`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
Review Comment:
Recommended to change the variable name to `segmentMap`, because `abort txn` is already in the class name
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ //Delete all segment
+ while (!abortTxnSegments.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ completableFutures.add(persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
Review Comment:
Deleting the `abortTxnSegments.firstKey` is running asynchronously, so there is a problem in the loop block: On the second loop, when we expect to delete the second element, the actual task is to delete the first element
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
Review Comment:
`lastSnapshotTimestamps` was also changed when doing `recover` or `delete segment`. So what does this variable really mean?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
Review Comment:
Maybe it is better to call this method `delete xxx`?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ //Delete all segment
+ while (!abortTxnSegments.isEmpty()) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ completableFutures.add(persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ //Delete index
+ return FutureUtil.waitForAll(completableFutures)
+ .thenCompose((ignore) -> snapshotIndexWriterFuture
+ .thenCompose(indexesWriter -> indexesWriter.writeAsync(topic.getName(), null)))
+ .thenRun(() -> {
+ log.info("Successes to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName());
+
+ })
+ .exceptionally(e -> {
+ log.error("Failed to clear the snapshot segment and indexes for the topic [{}]",
+ topic.getName(), e);
+
+ return null;
+ });
+ }
+
+
+ CompletableFuture<Void> closeAsync() {
Review Comment:
Should we cancel the timer task here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099609851
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
Review Comment:
It means all segments are read and handled. Maybe we should change `openManagedLedgerFuture` to `openManagedLedgerAndHandleSegmentsFuture`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1101459060
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,12 +55,17 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
this.topic = topic;
this.takeSnapshotWriter = this.topic.getBrokerService().getPulsar()
.getTransactionBufferSnapshotServiceFactory()
- .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()));
+ .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
Review Comment:
After this change, all operations will throw NPE if they use `takeSnapshotWriter`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102252251
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
| time | thread-`append task-write-segment` | thread-`execute task which in the queue` |
| --- | --- | --- |
| 1 | | write `segment-1` |
| 2 | | check: only has one task in the queue(`true`) |
| 3 | append new task `write-segment-2`, at this time there has two tasks in the queue | |
| 4 | clean unsealed list | |
| 5 | | update indexes |
if the broker crash after step 5: after reloading this topic, an error occurs: the maxReadPosition is the newest `maxReadPosition,` but segment 2 was lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102252251
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
| time | thread-`append task-write-segment` | thread-`execute task which in the queue` |
| --- | --- | --- |
| 1 | | write `segment-1` |
| 2 | | check: only has one task in the queue(`true`) |
| 3 | append new task `write-segment-2` | |
| 4 | clean unsealed list | |
| 5 | | update indexes |
if the broker crash after step 5: after reloading this topic, an error occurs: the maxReadPosition is the newest `maxReadPosition`, but segment 2 was lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095332335
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
Review Comment:
if the size is 10, how many aborted txn in one segment? I think ar-least 2 aborted txn in one segment
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
Review Comment:
if the size is 10, how many aborted txn in one segment? I think at-least 2 aborted txn in one segment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095347074
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
+ int theCountOfSnapshotMaxTxnCount = 3;
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSegmentedSnapshotEnabled(true);
+ this.getPulsarServiceList().get(0).getConfig().setTransactionBufferSnapshotSegmentSize(theSizeOfSegment);
+ this.getPulsarServiceList().get(0).getConfig()
+ .setTransactionBufferSnapshotMaxTransactionCount(theCountOfSnapshotMaxTxnCount);
+ // 1. Build producer and consumer
+ Producer<Integer> producer = pulsarClient.newProducer(Schema.INT32)
+ .topic(topic)
+ .enableBatching(false)
+ .create();
+
+ Consumer<Integer> consumer = pulsarClient.newConsumer(Schema.INT32)
+ .topic(topic)
+ .subscriptionName(subName)
+ .subscriptionType(SubscriptionType.Exclusive)
+ .subscribe();
+
+ // 2. Check the AbortedTxnProcessor workflow 10 times
+ int messageSize = theSizeOfSegment * 4;
+ for (int i = 0; i < 10; i++) {
+ MessageId maxReadMessage = null;
+ int abortedTxnSize = 0;
+ for (int j = 0; j < messageSize; j++) {
+ Transaction transaction = pulsarClient.newTransaction()
+ .withTransactionTimeout(5, TimeUnit.MINUTES).build().get();
+ //Half common message and half transaction message.
+ if (j % 2 == 0) {
+ MessageId messageId = producer.newMessage(transaction).value(i * 10 + j).send();
+ //And the transaction message have a half which are aborted.
+ if (RandomUtils.nextInt() % 2 == 0) {
+ transaction.abort().get();
+ abortedTxns.add(messageId);
+ abortedTxnSize++;
+ } else {
+ ongoingTxns.put(transaction, messageId);
+ if (maxReadMessage == null) {
+ //The except number of the messages that can be read
+ maxReadMessage = messageId;
+ }
+ }
+ } else {
+ producer.newMessage().value(i * 10 + j).send();
+ transaction.commit().get();
+ }
+ }
+ // 2.1 Receive all message before the maxReadPosition to verify the correctness of the max read position.
+ int hasReceived = 0;
+ while (true) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ if (message != null) {
+ Assert.assertTrue(message.getMessageId().compareTo(maxReadMessage) < 0);
+ hasReceived ++;
+ } else {
+ break;
+ }
+ }
+ //2.2 Commit all ongoing transaction and verify that the consumer can receive all rest message
+ // expect for aborted txn message.
+ for (Transaction ongoingTxn: ongoingTxns.keySet()) {
+ ongoingTxn.commit().get();
+ }
+ ongoingTxns.clear();
+ for (int k = hasReceived; k < messageSize - abortedTxnSize; k++) {
+ Message<Integer> message = consumer.receive(2, TimeUnit.SECONDS);
+ assertNotNull(message);
+ assertFalse(abortedTxns.contains(message.getMessageId()));
+ }
+ }
+ // 3. After the topic unload, the consumer can receive all the messages in the 10 tests
+ // expect for the aborted transaction messages.
Review Comment:
There is a clear snapshot test clearTransactionBufferSnapshotTest in TopicTransactionBufferRecoverTest.
And trim test in testPutAbortedTxnIntoProcessor.
```java
//3. Delete the ledgers and then verify the date.
Field ledgersField = ManagedLedgerImpl.class.getDeclaredField("ledgers");
ledgersField.setAccessible(true);
NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo> ledgers =
(NavigableMap<Long, MLDataFormats.ManagedLedgerInfo.LedgerInfo>)
ledgersField.get(persistentTopic.getManagedLedger());
ledgers.forEach((k, v) -> {
ledgers.remove(k);
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095811926
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -550,7 +558,7 @@ private void checkSnapshotCount(TopicName topicName, boolean hasSnapshot,
CompletableFuture<Long> compactionFuture = (CompletableFuture<Long>) field.get(persistentTopic);
Awaitility.await().untilAsserted(() -> assertTrue(compactionFuture.isDone()));
- Reader<TransactionBufferSnapshot> reader = pulsarClient.newReader(Schema.AVRO(TransactionBufferSnapshot.class))
+ Reader<GenericRecord> reader = pulsarClient.newReader(Schema.AUTO_CONSUME())
Review Comment:
we also need to check snapshot index topic right?
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -818,9 +820,98 @@ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Ob
//verify snapshot
assertEquals(snapshot.getTopicName(), snapshotTopic);
assertEquals(snapshot.getSequenceId(), 2L);
- assertEquals(snapshot.getMaxReadPositionLedgerId(), 2L);
- assertEquals(snapshot.getMaxReadPositionEntryId(), 3L);
+ assertEquals(snapshot.getPersistentPositionLedgerId(), 2L);
+ assertEquals(snapshot.getPersistentPositionEntryId(), 3L);
assertEquals(snapshot.getAborts().toArray()[0], new TxnIDData(1, 1));
}
+ //Verify the snapshotSegmentProcessor end to end
+ @Test
+ public void testSnapshotSegment() throws Exception {
+ String topic = NAMESPACE1 + "/testSnapshotSegment";
+ String subName = "testSnapshotSegment";
+
+ LinkedMap<Transaction, MessageId> ongoingTxns = new LinkedMap<>();
+ LinkedList<MessageId> abortedTxns = new LinkedList<>();
+ // 0. Modify the configurations, enabling the segment snapshot and set the size of the snapshot segment.
+ int theSizeOfSegment = 10;
Review Comment:
we should test at-least 2 aborted txn in a segment right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095811926
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TopicTransactionBufferRecoverTest.java:
##########
@@ -550,7 +558,7 @@ private void checkSnapshotCount(TopicName topicName, boolean hasSnapshot,
CompletableFuture<Long> compactionFuture = (CompletableFuture<Long>) field.get(persistentTopic);
Awaitility.await().untilAsserted(() -> assertTrue(compactionFuture.isDone()));
- Reader<TransactionBufferSnapshot> reader = pulsarClient.newReader(Schema.AVRO(TransactionBufferSnapshot.class))
+ Reader<GenericRecord> reader = pulsarClient.newReader(Schema.AUTO_CONSUME())
Review Comment:
we also need to check snapshot index topic has been clean right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058492478
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
Review Comment:
When deleting a Segment, we delete the Segment first and then update the Index. If the delete Segment succeeds, but the updated Index fails, then the error occurs here.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
Review Comment:
Create a delete task for each segment, and update the index within each delete task. This method costs too much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1067649532
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,651 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
Review Comment:
transactionBufferMaxAbortedTxnsOfSnapshotSegment is memory size, not number size
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,651 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, (cancel) ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ //The position is already deleted.
+ if (theLatestDeletedTxnID == null) {
+ continue;
+ }
Review Comment:
what time the theLatestDeletedTxnID can be null?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,651 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, (cancel) ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ //The position is already deleted.
+ if (theLatestDeletedTxnID == null) {
+ continue;
+ }
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ (cancel) -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList())));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, (cancel) -> {
+ if (!cancel) {
+ return persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ });
+ } else {
+ completableFuture.complete(null);
+ return completableFuture;
+ }
+ });
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.abortedMarkLedgerID,
+ transactionBufferSnapshotIndex.abortedMarkEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ //If the size of indexes is 0, the sequence ID will be the init value 0.
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ topic.close();
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ indexes.remove(new PositionImpl(
+ index.getAbortedMarkLedgerID(),
+ index.getAbortedMarkEntryID()));
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ //This is a compensation mechanism for deleting the segment
+ // but not successfully updating the index.
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, (cancel)
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ //Append the aborted txn IDs in the index metadata
+ //can keep the order of the aborted txn in the aborts.
+ //So that we can trim the expired snapshot segment in aborts
+ // according to the latest txn IDs
+ convertTypeToTxnID(persistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ (cancel) -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Function<Boolean, CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
Review Comment:
snapshotSegmentsWriterFuture create fail also should close topic
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,651 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
Review Comment:
codestyle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1081490507
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,707 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /**
+ * Calculate the segment capital according to its size configuration.
+ * <p>
+ * The empty transaction segment size is 5.
+ * Adding a empty linkedList, the size increase to 6.
+ * Add the topic name the size increase to the 7 + topic.getName().length().
+ * Add the aborted transaction IDs, the size increase to 8 +
+ * topic.getName().length() + 3 * aborted transaction ID size.
+ * </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /**
+ * The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ * Append a task to persistent the segment with the aborted transaction IDs and the latest
+ * transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /**
+ * Read the transaction snapshot segment index.
+ * <p>
+ * The processor can get the sequence ID, unsealed transaction IDs,
+ * segment index list and max read position in the snapshot segment index.
+ * Then we can traverse the index list to read all aborted transaction IDs
+ * in segments to aborts.
+ * </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ /**
+ * The logic flow of deleting expired segment is:
+ * <p>
+ * 1. delete segment
+ * 2. update segment index
+ * </p>
+ * If the worker delete segment successfully
+ * but failed to update segment index,
+ * the segment can not be read according to the index.
+ * We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /**
+ * Wait the processor recover completely and then allow TB
+ * to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /**
+ * Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /**
+ * If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /**
+ * Append the aborted txn IDs in the index metadata
+ * can keep the order of the aborted txn in the aborts.
+ * So that we can trim the expired snapshot segment in aborts
+ * according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /**
+ * The update index operation can be canceled when the task queue is not empty,
+ * so it should be executed immediately instead of appending to the task queue.
+ * If the taskQueue is not empty, the worker will execute the task in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /**
+ * Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ * The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /**
+ * Do not clear the snapshots if the topic is used.
+ * If the users want to delete a topic, he should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.TransactionBufferClosedException(
Review Comment:
return `ServiceUnitNotReadyException` is enough, don't add TransactionBufferClosedException
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
+ .NonRecoverableLedgerException) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl)topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
Review Comment:
I have fogoted, why we need executor to excute this task?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
Review Comment:
we don't need this flag, if don't have the index, the `startReadCursorPosition` and `persistentSnapshotIndexes` is null
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
Review Comment:
if the exceptin instanceof NonRecoverableLedgerException represent the indec is Invalid right?
```suggestion
if (!exception instanceof ManagedLedgerException
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1095338443
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,721 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, () -> persistentWorker
+ .updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ boolean hasIndex = false;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ if (exception instanceof ManagedLedgerException
Review Comment:
The ledger of the aborted mark is deleted meaning the index is invalid.
So we may delete the check for the exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1067173096
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,651 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Function;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its latest txn ID.
+ //This is mainly used to trim expired snapshot segment and remove them form aborts.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments whose key is the aborted mark persistent position.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, (cancel) ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(positionNeedToDelete);
+ //The position is already deleted.
+ if (theLatestDeletedTxnID == null) {
+ continue;
+ }
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(aborts.firstKey());
+ }
+ aborts.remove(theLatestDeletedTxnID);
+ }
+ //Batch delete the expired segment and then update segment index.
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ (cancel) -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete)
+ .thenRun(() -> persistentWorker.updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList())));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ return updateSnapshotIndexMetadata(metadata);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndexMetadata(TransactionBufferSnapshotIndexesMetadata metadata) {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, (cancel) -> {
+ if (!cancel) {
+ return persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
Review Comment:
this comment is outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1099973516
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,758 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata, indexes.values().stream().toList()));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(finalPersistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ convertTypeToTxnID(finalPersistentSnapshotIndexes.getSnapshot().getAborts())
+ .forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return CompletableFuture.completedFuture(null);
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ // update index after delete all segment.
+ private CompletableFuture<Void> deleteSnapshotSegment(List<PositionImpl> positionNeedToDeletes) {
+ List<CompletableFuture<Void>> results = new ArrayList<>();
+ for (PositionImpl positionNeedToDelete : positionNeedToDeletes) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ results.add(snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ //The process will check whether the snapshot segment is null,
+ // and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ }));
+ }
+ return FutureUtil.waitForAll(results);
+ }
+
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
Review Comment:
Yeap, fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1102288139
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
| time | thread-`TB.abortTxn` | thread-`execute task which in the queue` |
| --- | --- | --- |
| 1 | | write `segment-1` |
| 2 | update `maxReadPosition` of the topic | |
| 3 | | read `maxReadPosition` of the topic |
| 4 | | read `unsealedAborts` |
| 5 | put aborts to `unsealedAborts` | |
| 6 | | update indexes |
if the broker crash after step 6: after reloading this topic, an error occurs, we lost a record of txn-abort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1101070874
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,12 +55,17 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
this.topic = topic;
this.takeSnapshotWriter = this.topic.getBrokerService().getPulsar()
.getTransactionBufferSnapshotServiceFactory()
- .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()));
+ .getTxnBufferSnapshotService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
Review Comment:
Why should we catch ex here?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
Is there a scenario like this?
- The field `indexes` is `{aborts=[], maxReadPosition=-1:-1, segments=[] }`
- append task `write_segment_1` to the queue, the max read position is `3:3` now.
- append task `write_segment_2` to the queue, the max read position is `5:5` now.
- execute task `write_segment_1`
- update indexes to `[{aborts=[], maxReadPosition=5:5}, segments=[s1]]`
- broker crash
an error occurs: then segment 2 was lost, but the maxReadPosition of the topic is `5:5`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "liangyepianzhou (via GitHub)" <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1101334210
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,774 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.naming.SystemTopicNames;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ /**
+ * Stored the unsealed aborted transaction IDs Whose size is always less than the snapshotSegmentCapacity.
+ * It will be persistent as a snapshot segment when its size reach the configured capacity.
+ */
+ private LinkedList<TxnID> unsealedTxnIds;
+
+ /**
+ * The map is used to clear the aborted transaction IDs persistent in the expired ledger.
+ * <p>
+ * The key PositionImpl {@link PositionImpl} is the persistent position of
+ * the latest transaction of a segment.
+ * The value TxnID {@link TxnID} is the latest Transaction ID in a segment.
+ * </p>
+ *
+ * <p>
+ * If the position is expired, the processor can get the according latest
+ * transaction ID in this map. And then the processor can clear all the
+ * transaction IDs in the aborts {@link SnapshotSegmentAbortedTxnProcessorImpl#aborts}
+ * that lower than the transaction ID.
+ * And then the processor can delete the segments persistently according to
+ * the positions.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ /**
+ * This map is used to check whether a transaction is an aborted transaction.
+ * <p>
+ * The transaction IDs is appended in order, so the processor can delete expired
+ * transaction IDs according to the latest expired transaction IDs in segmentIndex
+ * {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}.
+ * </p>
+ */
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ /**
+ * This map stores the indexes of the snapshot segment.
+ * <p>
+ * The key is the persistent position of the marker of the last transaction in the segment.
+ * The value TransactionBufferSnapshotIndex {@link TransactionBufferSnapshotIndex} is the
+ * indexes of the snapshot segment.
+ * </p>
+ */
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ /**
+ * The number of the aborted transaction IDs in a segment.
+ * This is calculated according to the configured memory size.
+ */
+ private final int snapshotSegmentCapacity;
+ /**
+ * Responsible for executing the persistent tasks.
+ * <p>Including:</p>
+ * <p> Update segment index.</p>
+ * <p> Write snapshot segment.</p>
+ * <p> Delete snapshot segment.</p>
+ * <p> Clear all snapshot segment. </p>
+ */
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ /*
+ Calculate the segment capital according to its size configuration.
+ <p>
+ The empty transaction segment size is 5.
+ Adding an empty linkedList, the size increase to 6.
+ Add the topic name the size increase to the 7 + topic.getName().length().
+ Add the aborted transaction IDs, the size increase to 8 +
+ topic.getName().length() + 3 * aborted transaction ID size.
+ </p>
+ */
+ this.snapshotSegmentCapacity = (topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize() - 8 - topic.getName().length()) / 3;
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID txnID, PositionImpl position) {
+ unsealedTxnIds.add(txnID);
+ aborts.put(txnID, txnID);
+ /*
+ The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ Append a task to persistent the segment with the aborted transaction IDs and the latest
+ transaction mark persistent position passed by param.
+ */
+ if (unsealedTxnIds.size() >= snapshotSegmentCapacity) {
+ LinkedList<TxnID> abortedSegment = unsealedTxnIds;
+ segmentIndex.put(position, txnID);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment,
+ () -> persistentWorker.takeSnapshotSegmentAsync(abortedSegment, position));
+ this.unsealedTxnIds = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID) {
+ return aborts.containsKey(txnID);
+ }
+
+ /**
+ * Check werther the position in segmentIndex {@link SnapshotSegmentAbortedTxnProcessorImpl#segmentIndex}
+ * is expired. If the position is not exist in the original topic, the according transaction is an invalid
+ * transaction. And the according segment is invalid, too. The transaction IDs before the transaction ID
+ * in the aborts are invalid, too.
+ */
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+
+ TxnID theLatestDeletedTxnID = segmentIndex.remove(0);
+ while (!aborts.firstKey().equals(theLatestDeletedTxnID)) {
+ aborts.remove(0);
+ }
+ aborts.remove(0);
+ }
+ //Batch delete the expired segment
+ if (!positionsNeedToDelete.isEmpty()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ //Store the latest aborted transaction IDs in unsealedTxnIDs and the according the latest max read position.
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds));
+ return persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(metadata));
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ TransactionBufferSnapshotIndexes persistentSnapshotIndexes = null;
+ try {
+ /*
+ Read the transaction snapshot segment index.
+ <p>
+ The processor can get the sequence ID, unsealed transaction IDs,
+ segment index list and max read position in the snapshot segment index.
+ Then we can traverse the index list to read all aborted transaction IDs
+ in segments to aborts.
+ </p>
+ */
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ TransactionBufferSnapshotIndexes finalPersistentSnapshotIndexes = persistentSnapshotIndexes;
+ if (persistentSnapshotIndexes == null) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ this.unsealedTxnIds = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerAndHandleSegmentsFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ finalPersistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getSegmentLedgerID(),
+ index.getSegmentEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ handleSnapshotSegmentEntry(entry);
+ indexes.put(new PositionImpl(
+ index.abortedMarkLedgerID,
+ index.abortedMarkEntryID),
+ index);
+ entry.release();
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ /*
+ The logic flow of deleting expired segment is:
+ <p>
+ 1. delete segment
+ 2. update segment index
+ </p>
+ If the worker delete segment successfully
+ but failed to update segment index,
+ the segment can not be read according to the index.
+ We update index again if there are invalid indexes.
+ */
+ if (((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(index.getAbortedMarkLedgerID())) {
+ log.error("[{}] Failed to read snapshot segment [{}:{}]",
+ topic.getName(), index.segmentLedgerID,
+ index.segmentEntryID, exception);
+ handleSegmentFuture.completeExceptionally(exception);
+ } else {
+ hasInvalidIndex.set(true);
+ }
+ }
+ }, null);
+ });
+ openManagedLedgerAndHandleSegmentsFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerAndHandleSegmentsFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotSegmentTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ SystemTopicNames.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS);
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotSegmentTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ /*
+ Wait the processor recover completely and then allow TB
+ to recover the messages after the startReadCursorPosition.
+ */
+ return openManagedLedgerAndHandleSegmentsFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures))
+ .thenCompose((i) -> {
+ /*
+ Update the snapshot segment index if there exist invalid indexes.
+ */
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker.updateSnapshotIndex(
+ finalPersistentSnapshotIndexes.getSnapshot()));
+ }
+ /*
+ If there is no segment index, the persistent worker will write segment begin from 0.
+ */
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ /*
+ Append the aborted txn IDs in the index metadata
+ can keep the order of the aborted txn in the aborts.
+ So that we can trim the expired snapshot segment in aborts
+ according to the latest transaction IDs in the segmentIndex.
+ */
+ unsealedTxnIds.forEach(txnID -> aborts.put(txnID, txnID));
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ }).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> clearAbortedTxnSnapshot() {
+ return persistentWorker.appendTask(PersistentWorker.OperationType.Clear,
+ persistentWorker::clearSnapshotSegmentAndIndexes);
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+
+ TxnIDData lastTxn = snapshotSegment.getAborts().get(snapshotSegment.getAborts().size() - 1);
+ segmentIndex.put(new PositionImpl(snapshotSegment.getPersistentPositionLedgerId(),
+ snapshotSegment.getPersistentPositionEntryId()),
+ new TxnID(lastTxn.getMostSigBits(), lastTxn.getLeastSigBits()));
+ convertTypeToTxnID(snapshotSegment.getAborts()).forEach(txnID -> aborts.put(txnID, txnID));
+ }
+
+ private long getSystemClientOperationTimeoutMs() throws Exception {
+ PulsarClientImpl pulsarClient = (PulsarClientImpl) topic.getBrokerService().getPulsar().getClient();
+ return pulsarClient.getConfiguration().getOperationTimeoutMs();
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ /**
+ * The PersistentWorker be responsible for executing the persistent tasks, including:
+ * <p>
+ * 1. Write snapshot segment --- Encapsulate a sealed snapshot segment and persistent it.
+ * 2. Delete snapshot segment --- Evict expired snapshot segments.
+ * 3. Update snapshot indexes --- Update snapshot indexes after writing or deleting snapshot segment
+ * or update snapshot indexes metadata regularly.
+ * 4. Clear all snapshot segments and indexes. --- Executed when deleting this topic.
+ * </p>
+ * * Task 1 and task 2 will be put into a task queue. The tasks in the queue will be executed in order.
+ * * If the task queue is empty, task 3 will be executed immediately when it is appended to the worker.
+ * Else, the worker will try to execute the tasks in the task queue.
+ * * When task 4 was appended into worker, the worker will change the operation state to closed
+ * and cancel all tasks in the task queue. finally, execute the task 4 (clear task).
+ * If there are race conditions, throw an Exception to let users try again.
+ */
+ public class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ Operating,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Clear
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Pair<CompletableFuture<Void>,
+ Supplier<CompletableFuture<Void>>>>> taskQueue = new ConcurrentLinkedDeque<>();
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally(ex -> {
+ log.error("{} Failed to create snapshot index writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()))
+ .exceptionally((ex) -> {
+ log.error("{} Failed to create snapshot writer", topic.getName());
+ topic.close();
+ return null;
+ });
+ }
+
+ public CompletableFuture<Void> appendTask(OperationType operationType,
+ Supplier<CompletableFuture<Void>> task) {
+ CompletableFuture<Void> taskExecutedResult = new CompletableFuture<>();
+ switch (operationType) {
+ case UpdateIndex -> {
+ /*
+ The update index operation can be canceled when the task queue is not empty,
+ so it should be executed immediately instead of appending to the task queue.
+ If the taskQueue is not empty, the worker will execute the tasks in the queue.
+ */
+ if (!taskQueue.isEmpty()) {
+ executeTask();
+ return cancelUpdateIndexTask();
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ return task.get().whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating, OperationState.None);
+ });
+ } else {
+ return cancelUpdateIndexTask();
+ }
+ }
+ /*
+ Only the operations of WriteSegment and DeleteSegment will be appended into the taskQueue.
+ The operation will be canceled when the worker is close which means the topic is deleted.
+ */
+ case WriteSegment, DeleteSegment -> {
+ if (!STATE_UPDATER.get(this).equals(OperationState.Closed)) {
+ taskQueue.add(new MutablePair<>(operationType, new MutablePair<>(taskExecutedResult, task)));
+ executeTask();
+ return taskExecutedResult;
+ } else {
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+ case Clear -> {
+ /*
+ Do not clear the snapshots if the topic is used.
+ If the users want to delete a topic, they should stop the usage of the topic.
+ */
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Closed)) {
+ taskQueue.forEach(pair ->
+ pair.getRight().getRight().get().completeExceptionally(
+ new BrokerServiceException.ServiceUnitNotReadyException(
+ String.format("Cancel the operation [%s] due to the"
+ + " transaction buffer of the topic[%s] already closed",
+ pair.getLeft().name(), this.topic.getName()))));
+ taskQueue.clear();
+ /*
+ The task of clear all snapshot segments and indexes is executed immediately.
+ */
+ return task.get();
+ } else {
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.NotAllowedException(
+ String.format("Failed to clear the snapshot of topic [%s] due to "
+ + "the topic is used. Please stop the using of the topic "
+ + "and try it again", this.topic.getName())));
+ }
+ }
+ default -> {
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .NotAllowedException(String.format("Th operation [%s] is unsupported",
+ operationType.name())));
+ }
+ }
+ }
+
+ private CompletableFuture<Void> cancelUpdateIndexTask() {
+ if (log.isDebugEnabled()) {
+ log.debug("The operation of updating index is canceled due there is other operation executing");
+ }
+ return FutureUtil.failedFuture(new BrokerServiceException
+ .ServiceUnitNotReadyException("The operation of updating index is canceled"));
+ }
+
+ private void executeTask() {
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.Operating)) {
+ //Double-check. Avoid NoSuchElementException due to the first task is completed by other thread.
+ if (taskQueue.isEmpty()) {
+ return;
+ }
+ Pair<OperationType, Pair<CompletableFuture<Void>, Supplier<CompletableFuture<Void>>>> firstTask =
+ taskQueue.getFirst();
+ firstTask.getValue().getRight().get().whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to do operation do operation of [{}]",
+ topic.getName(), firstTask.getKey().name(), throwable);
+ }
+ firstTask.getRight().getKey().completeExceptionally(throwable);
+ } else {
+ firstTask.getRight().getKey().complete(null);
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this, OperationState.Operating,
+ OperationState.None);
+ //Execute the next task in the other thread.
+ topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this).submit(this::executeTask);
+ });
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(LinkedList<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ CompletableFuture<Void> res = writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment,
+ abortedMarkerPersistentPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ });
+ res.exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn next time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, abortedMarkerPersistentPosition, topic.getName(),
+ sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ return res;
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(LinkedList<TxnID> segment,
+ PositionImpl abortedMarkerPersistentPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(convertTypeToTxnIDData(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setPersistentPositionEntryId(abortedMarkerPersistentPosition.getEntryId());
+ transactionBufferSnapshotSegment.setPersistentPositionLedgerId(
+ abortedMarkerPersistentPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ PositionImpl maxReadPosition = topic.getMaxReadPosition();
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setAbortedMarkLedgerID(abortedMarkerPersistentPosition.getLedgerId());
+ index.setAbortedMarkEntryID(abortedMarkerPersistentPosition.getEntryId());
+ index.setSegmentLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setSegmentEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(abortedMarkerPersistentPosition, index);
+ //update snapshot segment index.
+ //If the index can not be written successfully, the snapshot segment wil be overwritten
+ // when the processor writes snapshot segment next time.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedTxnIds)));
Review Comment:
Fixed, I make the maxReadPosition is passed at appending write segment task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1058359302
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
Review Comment:
Let's instead `reader.readNext()` to `reader.readNextAsync().get()` and catch `timeout ex`, just like https://github.com/apache/pulsar/pull/18833
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,596 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, TxnID> segmentIndex = new ConcurrentSkipListMap<>();
+
+ private final LinkedList<TxnID> aborts = new LinkedList<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.contains(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
Review Comment:
I see that the variable `invalidIndex` is only used to identify whether the segment is valid or not, so why not use `boolean`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
poorbarcode commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060304860
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,599 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> abortTxnSegments =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ abortTxnSegments.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return abortTxnSegments.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ PositionImpl maxReadPosition = abortTxnSegments.ceilingKey((PositionImpl) readPosition);
+ if (maxReadPosition != null) {
+ return abortTxnSegments.get(maxReadPosition).contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!abortTxnSegments.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(abortTxnSegments.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), abortTxnSegments.firstKey());
+ }
+ PositionImpl positionNeedToDelete = abortTxnSegments.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ serializationForSegment(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNext();
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = deserializationFotSnapshotSegment(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.lastEntry().getValue().sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicLong invalidIndex = new AtomicLong(0);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ invalidIndex.getAndIncrement();
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (invalidIndex.get() != 0) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
+ () -> persistentWorker.clearSnapshotSegmentAndIndexes()
+ .thenRun(() -> {
+ completableFuture.thenCompose(null);
+ }).exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public long getLastSnapshotTimestamps() {
+ return this.lastSnapshotTimestamps;
+ }
+
+ @Override
+ public CompletableFuture<Void> closeAsync() {
+ return persistentWorker.closeAsync();
+ }
+
+ private void handleSnapshotSegmentEntry(Entry entry) {
+ //decode snapshot from entry
+ ByteBuf headersAndPayload = entry.getDataBuffer();
+ //skip metadata
+ Commands.parseMessageMetadata(headersAndPayload);
+ TransactionBufferSnapshotSegment snapshotSegment = Schema.AVRO(TransactionBufferSnapshotSegment.class)
+ .decode(Unpooled.wrappedBuffer(headersAndPayload).nioBuffer());
+ abortTxnSegments.put(new PositionImpl(snapshotSegment.getMaxReadPositionLedgerId(),
+ snapshotSegment.getMaxReadPositionEntryId()), deserializationFotSnapshotSegment(
+ snapshotSegment.getAborts()));
+
+ }
+
+ private <T> void closeReader(SystemTopicClient.Reader<T> reader) {
+ reader.closeAsync().exceptionally(e -> {
+ log.error("[{}]Transaction buffer snapshot reader close error!", topic.getName(), e);
+ return null;
+ });
+ }
+
+ private class PersistentWorker {
+ protected final AtomicLong sequenceID = new AtomicLong(0);
+
+ private final PersistentTopic topic;
+
+ //Persistent snapshot segment and index at the single thread.
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotSegment>>
+ snapshotSegmentsWriterFuture;
+ private final CompletableFuture<SystemTopicClient.Writer<TransactionBufferSnapshotIndexes>>
+ snapshotIndexWriterFuture;
+
+ private enum OperationState {
+ None,
+ UpdatingIndex,
+ WritingSegment,
+ DeletingSegment,
+ Closing,
+ Closed
+ }
+ private static final AtomicReferenceFieldUpdater<PersistentWorker, PersistentWorker.OperationState>
+ STATE_UPDATER = AtomicReferenceFieldUpdater.newUpdater(PersistentWorker.class,
+ PersistentWorker.OperationState.class, "operationState");
+
+ public enum OperationType {
+ UpdateIndex,
+ WriteSegment,
+ DeleteSegment,
+ Close
+ }
+
+ private volatile OperationState operationState = OperationState.None;
+
+ ConcurrentLinkedDeque<Pair<OperationType, Supplier<CompletableFuture<Void>>>> taskQueue =
+ new ConcurrentLinkedDeque<>();
+ private CompletableFuture<Void> lastOperationFuture;
+ private final Timer timer;
+
+ public PersistentWorker(PersistentTopic topic) {
+ this.topic = topic;
+ this.timer = topic.getBrokerService().getPulsar().getTransactionTimer();
+ this.snapshotSegmentsWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotSegmentService().createWriter(TopicName.get(topic.getName()));
+ this.snapshotIndexWriterFuture = this.topic.getBrokerService().getPulsar()
+ .getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService().createWriter(TopicName.get(topic.getName()));
+
+ }
+
+ public void appendTask(OperationType operationType, Supplier<CompletableFuture<Void>> task) {
+ switch (operationType) {
+ case UpdateIndex -> {
+ if (!taskQueue.isEmpty()) {
+ return;
+ } else if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.UpdatingIndex)) {
+ lastOperationFuture = task.get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null && log.isDebugEnabled()) {
+ log.debug("[{}] Failed to update index snapshot", topic.getName(), throwable);
+ }
+
+ STATE_UPDATER.compareAndSet(this, OperationState.UpdatingIndex, OperationState.None);
+ });
+ }
+ }
+ case WriteSegment, DeleteSegment -> {
+ taskQueue.add(new MutablePair<>(operationType, task));
+ executeTask();
+ }
+ case Close -> {
+ STATE_UPDATER.set(this, OperationState.Closing);
+ taskQueue.clear();
+ lastOperationFuture.thenRun(() -> {
+ task.get().thenRun(() ->
+ STATE_UPDATER.compareAndSet(this, OperationState.Closing, OperationState.Closed));
+ });
+ }
+ }
+ }
+
+ private void executeTask() {
+ OperationType operationType = taskQueue.getFirst().getKey();
+ switch (operationType) {
+ case WriteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.WritingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.WriteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to write snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.WritingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ case DeleteSegment -> {
+ if (STATE_UPDATER.compareAndSet(this, OperationState.None, OperationState.DeletingSegment)) {
+ if (taskQueue.getFirst().getKey() == OperationType.DeleteSegment) {
+ lastOperationFuture = taskQueue.getFirst().getValue().get();
+ lastOperationFuture.whenComplete((ignore, throwable) -> {
+ if (throwable != null) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Failed to delete snapshot segment", topic.getName(), throwable);
+ }
+ timer.newTimeout(timeout -> executeTask(),
+ takeSnapshotIntervalTime, TimeUnit.MILLISECONDS);
+ } else {
+ taskQueue.removeFirst();
+ }
+
+ STATE_UPDATER.compareAndSet(this,
+ OperationState.DeletingSegment, OperationState.None);
+ });
+ }
+ }
+ }
+ }
+ }
+
+ private CompletableFuture<Void> takeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> sealedAbortedTxnIdSegment,
+ PositionImpl maxReadPosition) {
+ return writeSnapshotSegmentAsync(sealedAbortedTxnIdSegment, maxReadPosition).thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("Successes to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size());
+ }
+ this.sequenceID.getAndIncrement();
+ }).exceptionally(e -> {
+ //Just log the error, and the processor will try to take snapshot again when the transactionBuffer
+ //append aborted txn nex time.
+ log.error("Failed to take snapshot segment [{}] at maxReadPosition [{}] "
+ + "for the topic [{}], and the size of the segment is [{}]",
+ this.sequenceID, maxReadPosition, topic.getName(), sealedAbortedTxnIdSegment.size(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> writeSnapshotSegmentAsync(ConcurrentOpenHashSet<TxnID> segment,
+ PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotSegment transactionBufferSnapshotSegment = new TransactionBufferSnapshotSegment();
+ transactionBufferSnapshotSegment.setAborts(serializationForSegment(segment));
+ transactionBufferSnapshotSegment.setTopicName(this.topic.getName());
+ transactionBufferSnapshotSegment.setMaxReadPositionEntryId(maxReadPosition.getEntryId());
+ transactionBufferSnapshotSegment.setMaxReadPositionLedgerId(maxReadPosition.getLedgerId());
+
+ return snapshotSegmentsWriterFuture.thenCompose(segmentWriter -> {
+ transactionBufferSnapshotSegment.setSequenceId(this.sequenceID.get());
+ return segmentWriter.writeAsync(buildKey(this.sequenceID.get()), transactionBufferSnapshotSegment);
+ }).thenCompose((messageId) -> {
+ //Build index for this segment
+ TransactionBufferSnapshotIndex index = new TransactionBufferSnapshotIndex();
+ index.setSequenceID(transactionBufferSnapshotSegment.getSequenceId());
+ index.setMaxReadPositionLedgerID(maxReadPosition.getLedgerId());
+ index.setMaxReadPositionEntryID(maxReadPosition.getEntryId());
+ index.setPersistentPositionLedgerID(((MessageIdImpl) messageId).getLedgerId());
+ index.setPersistentPositionEntryID(((MessageIdImpl) messageId).getEntryId());
+
+ indexes.put(maxReadPosition, index);
+ //update snapshot segment index.
+ return updateSnapshotIndex(new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(), new LinkedList<>()),
+ indexes.values().stream().toList());
+ });
+ }
+
+ private CompletableFuture<Void> deleteSnapshotSegment(PositionImpl positionNeedToDelete) {
+ long sequenceIdNeedToDelete = indexes.get(positionNeedToDelete).getSequenceID();
+ return snapshotSegmentsWriterFuture
+ .thenCompose(writer -> writer.deleteAsync(buildKey(sequenceIdNeedToDelete), null))
+ .thenRun(() -> {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Successes to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete);
+ }
+ abortTxnSegments.remove(positionNeedToDelete);
+ //The process will check whether the snapshot segment is null, and update index when recovered.
+ indexes.remove(positionNeedToDelete);
+ //Keep index snapshot and update index
+ updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList());
+ }).exceptionally(e -> {
+ log.warn("[{}] Failed to delete the snapshot segment, "
+ + "whose sequenceId is [{}] and maxReadPosition is [{}]",
+ this.topic.getName(), this.sequenceID, positionNeedToDelete, e);
+ return null;
+ });
+ }
+
+ //Update the indexes with the giving index snapshot and index list in the transactionBufferSnapshotIndexes.
+ private CompletableFuture<Void> updateSnapshotIndex(TransactionBufferSnapshotIndexesMetadata snapshotSegment,
+ List<TransactionBufferSnapshotIndex> indexList) {
+ TransactionBufferSnapshotIndexes snapshotIndexes = new TransactionBufferSnapshotIndexes();
+ return snapshotIndexWriterFuture
+ .thenCompose((indexesWriter) -> {
+ snapshotIndexes.setIndexList(indexList);
+ snapshotIndexes.setSnapshot(snapshotSegment);
+ return indexesWriter.writeAsync(topic.getName(), snapshotIndexes);
+ })
+ .thenRun(() -> {
+ persistentSnapshotIndexes = snapshotIndexes;
+ lastSnapshotTimestamps = System.currentTimeMillis();
+ })
+ .exceptionally(e -> {
+ log.error("[{}] Failed to update snapshot segment index", snapshotIndexes.getTopicName(), e);
+ return null;
+ });
+ }
+
+ private CompletableFuture<Void> clearSnapshotSegmentAndIndexes() {
Review Comment:
>> Create a delete task for each segment, and update the index within each delete task. This method costs too much.
> In factly, this logic in clearSnapshotSegmentAndIndexes is executed in a single task.
There does not open new tasks for each segment deleting and index updating.
Can we just batch-delete The Segments and only update The Indexes once?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060371440
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,614 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
Review Comment:
It is only operated at the beginning and end of the list, so I use it as a `LinkedList`.
If the size of unsealedAbortedTxnIdSegment is not large, `LinkedList` add faster than `ArrayList` due to the `ArrayList` need to expand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1060395101
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,618 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.collections4.map.LinkedMap;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.BrokerServiceException;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.client.impl.PulsarClientImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ //Stored the unsealed aborted txn id, it will be persistent as a snapshot segment and reinit
+ // when its size reach the capital of a snapshot segment.
+ private LinkedList<TxnID> unsealedAbortedTxnIdSegment;
+
+ //A mapping form the latest txn mark persistent position in a segment to its txn ID.
+ //This is mainly used to trim expired snapshot segment.
+ private final LinkedMap<PositionImpl, TxnID> segmentIndex = new LinkedMap<>();
+
+ //Store all aborted txn IDs check whether a txn is an aborted txn.
+ private final LinkedMap<TxnID, TxnID> aborts = new LinkedMap<>();
+ //The indexes of the snapshot segments.
+ private final LinkedMap<PositionImpl, TransactionBufferSnapshotIndex> indexes = new LinkedMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ private static final String SNAPSHOT_PREFIX = "multiple-";
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ aborts.put(abortedTxnId, abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ LinkedList<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentIndex.put(abortedMarkerPersistentPosition, abortedTxnId);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, abortedMarkerPersistentPosition));
+ this.unsealedAbortedTxnIdSegment = new LinkedList<>();
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ return aborts.containsKey(txnID);
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ List<PositionImpl> positionsNeedToDelete = new ArrayList<>();
+ while (!segmentIndex.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentIndex.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentIndex.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentIndex.firstKey();
+ positionsNeedToDelete.add(positionNeedToDelete);
+ }
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionsNeedToDelete));
+ }
+
+ private String buildKey(long sequenceId) {
+ return SNAPSHOT_PREFIX + sequenceId + "-" + this.topic.getName();
+ }
+
+ @Override
+ public CompletableFuture<Void> takeAbortedTxnsSnapshot(PositionImpl maxReadPosition) {
+ TransactionBufferSnapshotIndexesMetadata metadata = new TransactionBufferSnapshotIndexesMetadata(
+ maxReadPosition.getLedgerId(), maxReadPosition.getEntryId(),
+ convertTypeToTxnIDData(unsealedAbortedTxnIdSegment));
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex,
+ () -> persistentWorker
+ .updateSnapshotIndex(metadata, persistentSnapshotIndexes.getIndexList())
+ .thenRun(() -> completableFuture.complete(null))
+ .exceptionally(e -> {
+ completableFuture.completeExceptionally(e);
+ return null;
+ }));
+ return completableFuture;
+ }
+
+ @Override
+ public CompletableFuture<PositionImpl> recoverFromSnapshot() {
+ return topic.getBrokerService().getPulsar().getTransactionBufferSnapshotServiceFactory()
+ .getTxnBufferSnapshotIndexService()
+ .createReader(TopicName.get(topic.getName())).thenComposeAsync(reader -> {
+ PositionImpl startReadCursorPosition = null;
+ boolean hasIndex = false;
+ try {
+ //Read Index to recover the sequenceID, indexes, lastAbortedTxns and maxReadPosition.
+ while (reader.hasMoreEvents()) {
+ Message<TransactionBufferSnapshotIndexes> message = reader.readNextAsync()
+ .get(getSystemClientOperationTimeoutMs(), TimeUnit.MILLISECONDS);
+ if (topic.getName().equals(message.getKey())) {
+ TransactionBufferSnapshotIndexes transactionBufferSnapshotIndexes = message.getValue();
+ if (transactionBufferSnapshotIndexes != null) {
+ hasIndex = true;
+ this.persistentSnapshotIndexes = transactionBufferSnapshotIndexes;
+ startReadCursorPosition = PositionImpl.get(
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionLedgerId(),
+ transactionBufferSnapshotIndexes.getSnapshot().getMaxReadPositionEntryId());
+ }
+ }
+ }
+ } catch (TimeoutException ex) {
+ Throwable t = FutureUtil.unwrapCompletionException(ex);
+ String errorMessage = String.format("[%s] Transaction buffer recover fail by read "
+ + "transactionBufferSnapshot timeout!", topic.getName());
+ log.error(errorMessage, t);
+ return FutureUtil.failedFuture(
+ new BrokerServiceException.ServiceUnitNotReadyException(errorMessage, t));
+ } catch (Exception ex) {
+ log.error("[{}] Transaction buffer recover fail when read "
+ + "transactionBufferSnapshot!", topic.getName(), ex);
+ return FutureUtil.failedFuture(ex);
+ } finally {
+ closeReader(reader);
+ }
+ PositionImpl finalStartReadCursorPosition = startReadCursorPosition;
+ if (!hasIndex) {
+ return CompletableFuture.completedFuture(null);
+ } else {
+ persistentSnapshotIndexes.getIndexList()
+ .forEach(transactionBufferSnapshotIndex ->
+ indexes.put(new PositionImpl(
+ transactionBufferSnapshotIndex.persistentPositionLedgerID,
+ transactionBufferSnapshotIndex.persistentPositionEntryID),
+ transactionBufferSnapshotIndex));
+ this.unsealedAbortedTxnIdSegment = convertTypeToTxnID(persistentSnapshotIndexes
+ .getSnapshot().getAborts());
+ if (indexes.size() != 0) {
+ persistentWorker.sequenceID.set(indexes.get(indexes.lastKey()).sequenceID + 1);
+ }
+ }
+ //Read snapshot segment to recover aborts.
+ ArrayList<CompletableFuture<Void>> completableFutures = new ArrayList<>();
+ CompletableFuture<Void> openManagedLedgerFuture = new CompletableFuture<>();
+ AtomicBoolean hasInvalidIndex = new AtomicBoolean(false);
+ AsyncCallbacks.OpenReadOnlyManagedLedgerCallback callback = new AsyncCallbacks
+ .OpenReadOnlyManagedLedgerCallback() {
+ @Override
+ public void openReadOnlyManagedLedgerComplete(ReadOnlyManagedLedgerImpl readOnlyManagedLedger,
+ Object ctx) {
+ persistentSnapshotIndexes.getIndexList().forEach(index -> {
+ CompletableFuture<Void> handleSegmentFuture = new CompletableFuture<>();
+ completableFutures.add(handleSegmentFuture);
+ readOnlyManagedLedger.asyncReadEntry(
+ new PositionImpl(index.getPersistentPositionLedgerID(),
+ index.getPersistentPositionEntryID()),
+ new AsyncCallbacks.ReadEntryCallback() {
+ @Override
+ public void readEntryComplete(Entry entry, Object ctx) {
+ //Remove invalid index
+ if (entry == null) {
+ indexes.remove(new PositionImpl(
+ index.getMaxReadPositionLedgerID(),
+ index.getMaxReadPositionEntryID()));
+ handleSegmentFuture.complete(null);
+ hasInvalidIndex.set(true);
+ return;
+ }
+ handleSnapshotSegmentEntry(entry);
+ handleSegmentFuture.complete(null);
+ }
+
+ @Override
+ public void readEntryFailed(ManagedLedgerException exception, Object ctx) {
+ handleSegmentFuture.completeExceptionally(exception);
+ }
+ }, null);
+ });
+ openManagedLedgerFuture.complete(null);
+ }
+
+ @Override
+ public void openReadOnlyManagedLedgerFailed(ManagedLedgerException exception, Object ctx) {
+ log.error("[{}] Failed to open readOnly managed ledger", topic, exception);
+ openManagedLedgerFuture.completeExceptionally(exception);
+ }
+ };
+
+ TopicName snapshotIndexTopicName = TopicName.get(TopicDomain.persistent.toString(),
+ TopicName.get(topic.getName()).getNamespaceObject(),
+ EventType.TRANSACTION_BUFFER_SNAPSHOT_SEGMENTS.toString());
+ this.topic.getBrokerService().getPulsar().getManagedLedgerFactory()
+ .asyncOpenReadOnlyManagedLedger(snapshotIndexTopicName
+ .getPersistenceNamingEncoding(), callback,
+ topic.getManagedLedger().getConfig(),
+ null);
+ //Wait the processor recover completely and the allow TB to recover the messages
+ // after the startReadCursorPosition.
+
+ return openManagedLedgerFuture
+ .thenCompose((ignore) -> FutureUtil.waitForAll(completableFutures).thenCompose((i) -> {
+ if (hasInvalidIndex.get()) {
+ persistentWorker.appendTask(PersistentWorker.OperationType.UpdateIndex, ()
+ -> persistentWorker
+ .updateSnapshotIndex(persistentSnapshotIndexes.getSnapshot(),
+ indexes.values().stream().toList()));
+ }
+ return CompletableFuture.completedFuture(finalStartReadCursorPosition);
+ })).exceptionally(ex -> {
+ log.error("[{}] Failed to recover snapshot segment", this.topic.getName(), ex);
+ return null;
+ });
+
+ }, topic.getBrokerService().getPulsar().getTransactionExecutorProvider()
+ .getExecutor(this));
+ }
+
+ @Override
+ public CompletableFuture<Void> deleteAbortedTxnSnapshot() {
+ CompletableFuture<Void> completableFuture = new CompletableFuture<>();
+ persistentWorker.appendTask(PersistentWorker.OperationType.Close,
Review Comment:
Because this operation is executed in `transactionBufferCleanupAndClose`so I name it as close at the beginning.
But the `clear` may be better for the operation type of PersistentWorker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1052393968
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,600 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> segmentMap =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentMap.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return segmentMap.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
Review Comment:
can check the unsealed first, it will reduce checking the map. If check the map first, it may cost a lot of performance. If the consumption speed can keep up with the production speed, most of the txn check are in `unsealedAbortedTxnIdSegment`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,600 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> segmentMap =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentMap.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return segmentMap.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ Map.Entry<PositionImpl, ConcurrentOpenHashSet<TxnID>> cellingEntry =
+ segmentMap.ceilingEntry((PositionImpl) readPosition);
Review Comment:
abortedMarkerPersistentPosition compare to readPosition seem not correct right? maybe we should think about a good solution to implement it
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,600 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> segmentMap =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentMap.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return segmentMap.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ Map.Entry<PositionImpl, ConcurrentOpenHashSet<TxnID>> cellingEntry =
+ segmentMap.ceilingEntry((PositionImpl) readPosition);
+ if (cellingEntry != null) {
+ return cellingEntry.getValue().contains(txnID);
+ } else {
+ return unsealedAbortedTxnIdSegment.contains(txnID);
+ }
+ }
+ }
+
+ //In this implementation, we adopt snapshot segments. And then we clear invalid segment by its max read position.
+ @Override
+ public void trimExpiredAbortedTxns() {
+ //Checking whether there are some segment expired.
+ while (!segmentMap.isEmpty() && !((ManagedLedgerImpl) topic.getManagedLedger())
+ .ledgerExists(segmentMap.firstKey().getLedgerId())) {
+ if (log.isDebugEnabled()) {
+ log.debug("[{}] Topic transaction buffer clear aborted transactions, maxReadPosition : {}",
+ topic.getName(), segmentMap.firstKey());
+ }
+ PositionImpl positionNeedToDelete = segmentMap.firstKey();
+ persistentWorker.appendTask(PersistentWorker.OperationType.DeleteSegment,
+ () -> persistentWorker.deleteSnapshotSegment(positionNeedToDelete));
+ }
+ }
+
+ private String buildKey(long sequenceId) {
+ return "multiple-" + sequenceId + "-" + this.topic.getName();
Review Comment:
"multiple-" can be a final static field
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SingleSnapshotAbortedTxnProcessorImpl.java:
##########
@@ -56,8 +56,8 @@ public SingleSnapshotAbortedTxnProcessorImpl(PersistentTopic topic) {
}
@Override
- public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl position) {
- aborts.put(abortedTxnId, position);
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ aborts.put(abortedTxnId, this.topic.getMaxReadPosition());
Review Comment:
why change the position to this.topic.getMaxReadPosition()?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on a diff in pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on code in PR #18273:
URL: https://github.com/apache/pulsar/pull/18273#discussion_r1052886590
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/buffer/impl/SnapshotSegmentAbortedTxnProcessorImpl.java:
##########
@@ -0,0 +1,600 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,2
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.transaction.buffer.impl;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.Unpooled;
+import io.netty.util.Timer;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;
+import java.util.function.Supplier;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.bookkeeper.mledger.AsyncCallbacks;
+import org.apache.bookkeeper.mledger.Entry;
+import org.apache.bookkeeper.mledger.ManagedLedgerException;
+import org.apache.bookkeeper.mledger.Position;
+import org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl;
+import org.apache.bookkeeper.mledger.impl.PositionImpl;
+import org.apache.bookkeeper.mledger.impl.ReadOnlyManagedLedgerImpl;
+import org.apache.commons.lang3.tuple.MutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pulsar.broker.service.persistent.PersistentTopic;
+import org.apache.pulsar.broker.systopic.SystemTopicClient;
+import org.apache.pulsar.broker.transaction.buffer.AbortedTxnProcessor;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndex;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexes;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotIndexesMetadata;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TransactionBufferSnapshotSegment;
+import org.apache.pulsar.broker.transaction.buffer.metadata.v2.TxnIDData;
+import org.apache.pulsar.client.api.Message;
+import org.apache.pulsar.client.api.Schema;
+import org.apache.pulsar.client.api.transaction.TxnID;
+import org.apache.pulsar.client.impl.MessageIdImpl;
+import org.apache.pulsar.common.events.EventType;
+import org.apache.pulsar.common.naming.TopicDomain;
+import org.apache.pulsar.common.naming.TopicName;
+import org.apache.pulsar.common.protocol.Commands;
+import org.apache.pulsar.common.util.FutureUtil;
+import org.apache.pulsar.common.util.collections.ConcurrentOpenHashSet;
+
+@Slf4j
+public class SnapshotSegmentAbortedTxnProcessorImpl implements AbortedTxnProcessor {
+
+ private ConcurrentOpenHashSet<TxnID> unsealedAbortedTxnIdSegment;
+
+ //Store the fixed aborted transaction segment
+ private final ConcurrentSkipListMap<PositionImpl, ConcurrentOpenHashSet<TxnID>> segmentMap =
+ new ConcurrentSkipListMap<>();
+
+ private final ConcurrentSkipListMap<PositionImpl, TransactionBufferSnapshotIndex> indexes =
+ new ConcurrentSkipListMap<>();
+ //The latest persistent snapshot index. This is used to combine new segment indexes with the latest metadata and
+ // indexes.
+ private TransactionBufferSnapshotIndexes persistentSnapshotIndexes = new TransactionBufferSnapshotIndexes();
+
+ private final PersistentTopic topic;
+
+ private volatile long lastSnapshotTimestamps;
+
+ private final int takeSnapshotIntervalTime;
+
+ private final int transactionBufferMaxAbortedTxnsOfSnapshotSegment;
+ private final PersistentWorker persistentWorker;
+
+ public SnapshotSegmentAbortedTxnProcessorImpl(PersistentTopic topic) {
+ this.topic = topic;
+ this.persistentWorker = new PersistentWorker(topic);
+ this.takeSnapshotIntervalTime = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotMinTimeInMillis();
+ this.transactionBufferMaxAbortedTxnsOfSnapshotSegment = topic.getBrokerService().getPulsar()
+ .getConfiguration().getTransactionBufferSnapshotSegmentSize();
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+
+ @Override
+ public void putAbortedTxnAndPosition(TxnID abortedTxnId, PositionImpl abortedMarkerPersistentPosition) {
+ unsealedAbortedTxnIdSegment.add(abortedTxnId);
+ //The size of lastAbortedTxns reaches the configuration of the size of snapshot segment.
+ if (unsealedAbortedTxnIdSegment.size() == transactionBufferMaxAbortedTxnsOfSnapshotSegment) {
+ ConcurrentOpenHashSet<TxnID> abortedSegment = unsealedAbortedTxnIdSegment;
+ segmentMap.put(abortedMarkerPersistentPosition, abortedSegment);
+ persistentWorker.appendTask(PersistentWorker.OperationType.WriteSegment, () ->
+ persistentWorker.takeSnapshotSegmentAsync(abortedSegment, this.topic.getMaxReadPosition()));
+ this.unsealedAbortedTxnIdSegment =
+ new ConcurrentOpenHashSet<>(this.transactionBufferMaxAbortedTxnsOfSnapshotSegment, 1);
+ }
+ }
+
+ @Override
+ public boolean checkAbortedTransaction(TxnID txnID, Position readPosition) {
+ if (readPosition == null) {
+ return segmentMap.values().stream()
+ .anyMatch(list -> list.contains(txnID)) || unsealedAbortedTxnIdSegment.contains(txnID);
+ } else {
+ Map.Entry<PositionImpl, ConcurrentOpenHashSet<TxnID>> cellingEntry =
+ segmentMap.ceilingEntry((PositionImpl) readPosition);
Review Comment:
Why can not compare abortedMarkerPersistentPosition and read position?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 merged pull request #18273: [feat][txn] implement the SnapshotSegmentAbortedTxnProcessor
Posted by "congbobo184 (via GitHub)" <gi...@apache.org>.
congbobo184 merged PR #18273:
URL: https://github.com/apache/pulsar/pull/18273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org