You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ratis.apache.org by ta...@apache.org on 2024/11/18 04:36:25 UTC
(ratis) branch snapshot-3 updated (af6438ac2 -> 2fe59d3f1)
This is an automated email from the ASF dual-hosted git repository.
tanxinyu pushed a change to branch snapshot-3
in repository https://gitbox.apache.org/repos/asf/ratis.git
discard af6438ac2 RATIS-2178. Add a log message whenever deleting a RaftLog segment. (#1170)
discard 87ba60c59 RATIS-2179. Fix NPE in RaftServerImpl.getLogInfo. (#1171)
discard 7430a65bd release 3.1.2-5cd8971-SNAPSHOT
discard 5cd8971b6 RATIS-2174. Move future.join outside the lock (#1168)
discard 246b7eac1 release 3.1.2-b09bfe7-SNAPSHOT
discard b09bfe771 RATIS-2172. RaftServer may lose FollowerState (#1166)
discard 9f77cddc4 release 3.1.2-536419c-SNAPSHOT
add fbb9b5a66 RATIS-2172. RaftServer may lose FollowerState (#1166)
add 7f10888a4 RATIS-2174. Move future.join outside the lock (#1168)
add 263ea69b2 RATIS-2179. Fix NPE in RaftServerImpl.getLogInfo. (#1171)
add 2981c6b15 RATIS-2178. Add a log message whenever deleting a RaftLog segment. (#1170)
add 7c3942d1d RATIS-2183. Detect staled snapshot request. (#1173)
add ebc9c3e65 Change version for the version 3.1.2 -rc1
new 5ed4341b0 RATIS-2177. Purge should delete segmentLog from small to large according to logIndex (#1174)
new 2fe59d3f1 RATIS-2186. Raft log should not purge index lower than the log start index (#1175)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (af6438ac2)
\
N -- N -- N refs/heads/snapshot-3 (2fe59d3f1)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
pom.xml | 4 +--
ratis-assembly/pom.xml | 2 +-
ratis-client/pom.xml | 2 +-
ratis-common/pom.xml | 2 +-
ratis-docs/pom.xml | 2 +-
ratis-examples/pom.xml | 2 +-
ratis-experiments/pom.xml | 2 +-
ratis-grpc/pom.xml | 2 +-
.../apache/ratis/grpc/server/GrpcLogAppender.java | 2 ++
ratis-metrics-api/pom.xml | 2 +-
ratis-metrics-default/pom.xml | 2 +-
ratis-metrics-dropwizard3/pom.xml | 2 +-
ratis-netty/pom.xml | 2 +-
ratis-proto/pom.xml | 2 +-
ratis-proto/src/main/proto/Raft.proto | 1 +
ratis-replicated-map/pom.xml | 2 +-
ratis-resource-bundle/pom.xml | 2 +-
ratis-server-api/pom.xml | 2 +-
ratis-server/pom.xml | 2 +-
.../server/impl/SnapshotInstallationHandler.java | 17 ++++++++++++
.../ratis/server/leader/LogAppenderDefault.java | 1 +
.../apache/ratis/server/raftlog/RaftLogBase.java | 20 ++++++++++-----
.../raftlog/segmented/SegmentedRaftLogCache.java | 6 ++++-
.../raftlog/segmented/SegmentedRaftLogWorker.java | 7 ++---
ratis-shell/pom.xml | 2 +-
ratis-test/pom.xml | 2 +-
.../raftlog/segmented/TestSegmentedRaftLog.java | 30 +++++++++++++++++++---
ratis-tools/pom.xml | 2 +-
28 files changed, 91 insertions(+), 35 deletions(-)
(ratis) 02/02: RATIS-2186. Raft log should not purge index lower than the log start index (#1175)
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tanxinyu pushed a commit to branch snapshot-3
in repository https://gitbox.apache.org/repos/asf/ratis.git
commit 2fe59d3f14fd6cfd07f198896969640a3f107787
Author: Ivan Andika <iv...@rocketmail.com>
AuthorDate: Wed Nov 13 00:23:04 2024 +0800
RATIS-2186. Raft log should not purge index lower than the log start index (#1175)
---
.../apache/ratis/server/raftlog/RaftLogBase.java | 20 ++++++++++-----
.../raftlog/segmented/SegmentedRaftLogCache.java | 6 ++++-
.../raftlog/segmented/TestSegmentedRaftLog.java | 30 +++++++++++++++++++---
3 files changed, 45 insertions(+), 11 deletions(-)
diff --git a/ratis-server/src/main/java/org/apache/ratis/server/raftlog/RaftLogBase.java b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/RaftLogBase.java
index 1e22e07bd..9b0367213 100644
--- a/ratis-server/src/main/java/org/apache/ratis/server/raftlog/RaftLogBase.java
+++ b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/RaftLogBase.java
@@ -314,20 +314,28 @@ public abstract class RaftLogBase implements RaftLog {
@Override
public final CompletableFuture<Long> purge(long suggestedIndex) {
+ final long adjustedIndex;
if (purgePreservation > 0) {
final long currentIndex = getNextIndex() - 1;
- suggestedIndex = Math.min(suggestedIndex, currentIndex - purgePreservation);
+ adjustedIndex = Math.min(suggestedIndex, currentIndex - purgePreservation);
+ } else {
+ adjustedIndex = suggestedIndex;
}
final long lastPurge = purgeIndex.get();
- if (suggestedIndex - lastPurge < purgeGap) {
+ if (adjustedIndex - lastPurge < purgeGap) {
+ return CompletableFuture.completedFuture(lastPurge);
+ }
+ final long startIndex = getStartIndex();
+ if (adjustedIndex < startIndex) {
+ LOG.info("{}: purge({}) is skipped: adjustedIndex = {} < startIndex = {}, purgePreservation = {}",
+ getName(), suggestedIndex, adjustedIndex, startIndex, purgePreservation);
return CompletableFuture.completedFuture(lastPurge);
}
- LOG.info("{}: purge {}", getName(), suggestedIndex);
- final long finalSuggestedIndex = suggestedIndex;
- return purgeImpl(suggestedIndex).whenComplete((purged, e) -> {
+ LOG.info("{}: purge {}", getName(), adjustedIndex );
+ return purgeImpl(adjustedIndex).whenComplete((purged, e) -> {
updatePurgeIndex(purged);
if (e != null) {
- LOG.warn(getName() + ": Failed to purge " + finalSuggestedIndex, e);
+ LOG.warn(getName() + ": Failed to purge " + adjustedIndex, e);
}
});
}
diff --git a/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogCache.java b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogCache.java
index 20b7a5c37..9509436c3 100644
--- a/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogCache.java
+++ b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogCache.java
@@ -356,7 +356,11 @@ public class SegmentedRaftLogCache {
TruncationSegments purge(long index) {
try (AutoCloseableLock writeLock = writeLock()) {
int segmentIndex = binarySearch(index);
- List<SegmentFileInfo> list = new ArrayList<>();
+ if (segmentIndex == -1) {
+ // nothing to purge
+ return null;
+ }
+ List<SegmentFileInfo> list = new LinkedList<>();
if (segmentIndex == -segments.size() - 1) {
for (LogSegment ls : segments) {
diff --git a/ratis-test/src/test/java/org/apache/ratis/server/raftlog/segmented/TestSegmentedRaftLog.java b/ratis-test/src/test/java/org/apache/ratis/server/raftlog/segmented/TestSegmentedRaftLog.java
index 93eb7db0e..a772b0002 100644
--- a/ratis-test/src/test/java/org/apache/ratis/server/raftlog/segmented/TestSegmentedRaftLog.java
+++ b/ratis-test/src/test/java/org/apache/ratis/server/raftlog/segmented/TestSegmentedRaftLog.java
@@ -563,6 +563,7 @@ public class TestSegmentedRaftLog extends BaseTest {
int segmentSize = 200;
long beginIndexOfOpenSegment = segmentSize * (endTerm - startTerm - 1);
long expectedIndex = segmentSize * (endTerm - startTerm - 1);
+ long purgePreservation = 0L;
purgeAndVerify(startTerm, endTerm, segmentSize, 1, beginIndexOfOpenSegment, expectedIndex);
}
@@ -599,15 +600,36 @@ public class TestSegmentedRaftLog extends BaseTest {
purgeAndVerify(startTerm, endTerm, segmentSize, 1000, endIndexOfClosedSegment, expectedIndex);
}
+ @Test
+ public void testPurgeWithLargePurgePreservationAndSmallPurgeGap() throws Exception {
+ int startTerm = 0;
+ int endTerm = 5;
+ int segmentSize = 200;
+ long endIndex = segmentSize * (endTerm - startTerm) - 1;
+ // start index is set so that the suggested index will not be negative, which will not trigger any purge
+ long startIndex = 200;
+ // purge preservation is larger than the total size of the log entries
+ // which causes suggested index to be lower than the start index
+ long purgePreservation = (segmentSize * (endTerm - startTerm )) + 100;
+ // if the suggested index is lower than the start index due to the purge preservation, we should not purge anything
+ purgeAndVerify(startTerm, endTerm, segmentSize, 1, endIndex, startIndex, startIndex, purgePreservation);
+ }
+
private void purgeAndVerify(int startTerm, int endTerm, int segmentSize, int purgeGap, long purgeIndex,
- long expectedIndex) throws Exception {
- List<SegmentRange> ranges = prepareRanges(startTerm, endTerm, segmentSize, 0);
+ long expectedIndex) throws Exception {
+ purgeAndVerify(startTerm, endTerm, segmentSize, purgeGap, purgeIndex, expectedIndex, 0, 0);
+ }
+
+ private void purgeAndVerify(int startTerm, int endTerm, int segmentSize, int purgeGap, long purgeIndex,
+ long expectedIndex, long startIndex, long purgePreservation) throws Exception {
+ List<SegmentRange> ranges = prepareRanges(startTerm, endTerm, segmentSize, startIndex);
List<LogEntryProto> entries = prepareLogEntries(ranges, null);
final RaftProperties p = new RaftProperties();
RaftServerConfigKeys.Log.setPurgeGap(p, purgeGap);
- try (SegmentedRaftLog raftLog = newSegmentedRaftLog(storage, p)) {
- raftLog.open(RaftLog.INVALID_LOG_INDEX, null);
+ RaftServerConfigKeys.Log.setPurgePreservationLogNum(p, purgePreservation);
+ try (SegmentedRaftLog raftLog = newSegmentedRaftLogWithSnapshotIndex(storage, p, () -> startIndex - 1)) {
+ raftLog.open(startIndex - 1, null);
entries.stream().map(raftLog::appendEntry).forEach(CompletableFuture::join);
final CompletableFuture<Long> f = raftLog.purge(purgeIndex);
final Long purged = f.get();
(ratis) 01/02: RATIS-2177. Purge should delete segmentLog from small to large according to logIndex (#1174)
Posted by ta...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tanxinyu pushed a commit to branch snapshot-3
in repository https://gitbox.apache.org/repos/asf/ratis.git
commit 5ed4341b054f2c5c8e2ce0f4be9f909c19ec892d
Author: 133tosakarin <97...@users.noreply.github.com>
AuthorDate: Sun Nov 10 10:17:29 2024 +0800
RATIS-2177. Purge should delete segmentLog from small to large according to logIndex (#1174)
---
.../ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java
index a8482e71b..c59c37be8 100644
--- a/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java
+++ b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java
@@ -476,9 +476,10 @@ class SegmentedRaftLogWorker {
void execute() throws IOException {
if (segments.getToDelete() != null) {
try(UncheckedAutoCloseable ignored = raftLogMetrics.startPurgeTimer()) {
- for (SegmentFileInfo fileInfo : segments.getToDelete()) {
- final Path deleted = FileUtils.deleteFile(fileInfo.getFile(storage));
- LOG.info("{}: Purged RaftLog segment: info={}, path={}", name, fileInfo, deleted);
+ SegmentFileInfo[] toDeletes = segments.getToDelete();
+ for (int i = toDeletes.length - 1; i >= 0; i--) {
+ final Path deleted = FileUtils.deleteFile(toDeletes[i].getFile(storage));
+ LOG.info("{}: Purged RaftLog segment: info={}, path={}", name, toDeletes[i], deleted);
}
}
}