You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "xleoken (via GitHub)" <gi...@apache.org> on 2023/10/10 02:19:26 UTC
[PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
xleoken opened a new pull request, #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966
…ng SimpleStateMachineStorage
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
We need to delete md5 file init SimpleStateMachineStorage based on ratis-2.0.0, but the logic about cleanup md5 files already support after RATIS-1752, so we can optimize initialization.
https://issues.apache.org/jira/browse/RATIS-1752
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1361591209
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
Hi @xleoken
After I checked the old logic, there shouldn't be md5 files left over and they will be cleaned. As we can see the old logic deletes snapshot files in two steps.
1. Find all snapshot files and delete them with the md5 files.
2. Find all md5 files without the corresponding snapshot files and delete them.
```java
try (DirectoryStream<Path> stream =
Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
for (Path path : stream) {
if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
continue;
} else {
filePatternMatches(MD5_REGEX, allMD5Files, path);
}
}
}
// first step, cleanup old snapshot and md5 file
SingleFileSnapshotInfo snapshotInfo =
cleanupOldFiles(
allSnapshotFiles,
snapshotRetentionPolicy.getNumSnapshotsRetained(),
false,
null);
// second step, cleanup only old md5 file
cleanupOldFiles(
allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
```
Have you found out why ratis will left with md5 files only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1360336179
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
BTW, is this old logic correct? IIUC when `onlyCleanupMD5Files` is true, we should delete the md5 file instead of the original file. cc @FMX @RexXiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1362026928
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
It's by design to keep the number of `snapshotRetentionPolicy.getNumSnapshotsRetained()` files. Ratis won't create MD5 only.
Check the codes below.
```java
private SingleFileSnapshotInfo cleanupOldFiles(
List<SingleFileSnapshotInfo> inputFiles,
int retainedNum,
boolean onlyCleanupMD5Files,
SingleFileSnapshotInfo snapshotInfo) {
SingleFileSnapshotInfo result = null;
if (inputFiles.size() > retainedNum) {
inputFiles.sort(new RatisSnapshotFileComparator());
List<SingleFileSnapshotInfo> filesToBeCleaned =
inputFiles.subList(retainedNum, inputFiles.size());
result = filesToBeCleaned.get(0);
for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
|| (onlyCleanupMD5Files && null == snapshotInfo))) {
continue;
}
File file = fileInfo.getFile().getPath().toFile();
if (onlyCleanupMD5Files) {
LOG.info("Deleting old md5 file at {}.", file.getAbsolutePath());
FileUtils.deleteFileQuietly(file);
} else {
File md5File = new File(file.getAbsolutePath() + MD5FileUtil.MD5_SUFFIX);
LOG.info(
"Deleting old snapshot at {}, md5 file at {}.",
file.getAbsolutePath(),
md5File.getAbsolutePath());
FileUtils.deleteFileQuietly(file);
FileUtils.deleteFileQuietly(md5File);
}
}
}
return result;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#issuecomment-1758823752
Hi @FMX, help to take a review again, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #1966: [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage
URL: https://github.com/apache/incubator-celeborn/pull/1966
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1361404226
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
Let me give you one example (follow the old logic), we take 5 snapshots and set the numSnapshotsRetained to 3.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1365067081
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
> Due to the Ratis community is not very active, the old logic has been verified for a while. Can you verify this PR in your test environment?
@FMX @waitinfuture take a look again when free.
Yeah, I startup a celeborn cluster(3 master with 1 worker), and run some shuffle examples.
And let `raft.server.snapshot.retention.file.num` to `3`, after tests, obtained the expected results.
**Conf**
```
celeborn.master.ha.ratis.raft.server.storage.dir /opt/data/celeborn_ratis/
celeborn.master.ha.ratis.raft.server.snapshot.auto.trigger.enabled true
celeborn.master.ha.ratis.raft.server.snapshot.auto.trigger.threshold 1
celeborn.master.ha.ratis.raft.server.snapshot.retention.file.num 3
```
**Snapshots**
```
root@kaaa:/opt/data#
root@kaaa:/opt/data# curl 172.17.0.3:9098/masterGroupInfo
====================== Master Group INFO ==============================
group id: c5196f6d-2c34-3ed3-8b8a-47bede733167
leader info: 1(kaaa.hadoop.com:9872)
[server {
id: "2"
address: "172.17.0.3:9872"
clientAddress: "172.17.0.3:9097"
startupRole: FOLLOWER
}
commitIndex: 148
, server {
id: "1"
address: "kaaa.hadoop.com:9872"
clientAddress: "172.17.0.2:9097"
startupRole: FOLLOWER
}
commitIndex: 148
, server {
id: "3"
address: "172.17.0.4:9872"
clientAddress: "172.17.0.4:9097"
startupRole: FOLLOWER
}
commitIndex: 148
]
root@kaaa:/opt/data# ll /opt/data/celeborn_ratis/c5196f6d-2c34-3ed3-8b8a-47bede733167/sm
total 24
drwxr-xr-x 2 root root 150 Oct 19 07:54 ./
drwxr-xr-x 5 root root 61 Oct 19 07:31 ../
-rw-r--r-- 1 root root 205 Oct 19 07:54 snapshot.1_144
-rw-r--r-- 1 root root 49 Oct 19 07:54 snapshot.1_144.md5
-rw-r--r-- 1 root root 205 Oct 19 07:54 snapshot.1_146
-rw-r--r-- 1 root root 49 Oct 19 07:54 snapshot.1_146.md5
-rw-r--r-- 1 root root 205 Oct 19 07:54 snapshot.1_148
-rw-r--r-- 1 root root 49 Oct 19 07:54 snapshot.1_148.md5
root@kaaa:/opt/data#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1360463688
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
the old logic is wrong here,(compared to `ratis-2.5.1`), we also need clean up the md5 files if the corresponding snapshot file does not exist.
https://github.com/apache/ratis/blob/77640a3a1d012d0225d26a5a52b38fb80bb9fe44/ratis-server/src/main/java/org/apache/ratis/statemachine/impl/SimpleStateMachineStorage.java#L129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1365067081
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
> Due to the Ratis community is not very active, the old logic has been verified for a while. Can you verify this PR in your test environment?
@FMX @waitinfuture take a look again when free.
Yeah, I run a celeborn cluster(3 master with 1 worker), and run some shuffle examples.
And let `raft.server.snapshot.retention.file.num` to `3`, after tests, obtained the expected results.
**Conf**
```
celeborn.master.ha.ratis.raft.server.storage.dir /opt/data/celeborn_ratis/
celeborn.master.ha.ratis.raft.server.snapshot.auto.trigger.enabled true
celeborn.master.ha.ratis.raft.server.snapshot.auto.trigger.threshold 1
celeborn.master.ha.ratis.raft.server.snapshot.retention.file.num 3
```
**Snapshots**
```
root@kaaa:/opt/data#
root@kaaa:/opt/data# curl 172.17.0.3:9098/masterGroupInfo
====================== Master Group INFO ==============================
group id: c5196f6d-2c34-3ed3-8b8a-47bede733167
leader info: 1(kaaa.hadoop.com:9872)
[server {
id: "2"
address: "172.17.0.3:9872"
clientAddress: "172.17.0.3:9097"
startupRole: FOLLOWER
}
commitIndex: 82
, server {
id: "1"
address: "kaaa.hadoop.com:9872"
clientAddress: "172.17.0.2:9097"
startupRole: FOLLOWER
}
commitIndex: 82
, server {
id: "3"
address: "172.17.0.4:9872"
clientAddress: "172.17.0.4:9097"
startupRole: FOLLOWER
}
commitIndex: 82
]
root@kaaa:/opt/data#
root@kaaa:/opt/data#
root@kaaa:/opt/data# ll /opt/data/celeborn_ratis/c5196f6d-2c34-3ed3-8b8a-47bede733167/sm
total 24
drwxr-xr-x 2 root root 144 Oct 19 07:50 ./
drwxr-xr-x 5 root root 61 Oct 19 07:31 ../
-rw-r--r-- 1 root root 175 Oct 19 07:50 snapshot.1_80
-rw-r--r-- 1 root root 48 Oct 19 07:50 snapshot.1_80.md5
-rw-r--r-- 1 root root 175 Oct 19 07:50 snapshot.1_82
-rw-r--r-- 1 root root 48 Oct 19 07:50 snapshot.1_82.md5
-rw-r--r-- 1 root root 175 Oct 19 07:50 snapshot.1_84
-rw-r--r-- 1 root root 48 Oct 19 07:50 snapshot.1_84.md5
root@kaaa:/opt/data#
root@kaaa:/opt/data#
root@kaaa:/opt/data#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#issuecomment-1755249369
cc @FMX could you take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#issuecomment-1755253724
> cc @FMX could you take a look at this PR?
OK, I'll review and check this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1362070789
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
Due to the Ratis community is not very active, the old logic has been verified for a while. Can you verify this PR in your test environment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1361591209
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
Hi @xleoken
After I checked the old logic, there should be md5 files be left over and they will be cleaned. As we can see that the old logic delete snapshot files in two steps.
1. Find all snapshot files and delete them with the md5 files.
2. Find all md5 files without the corresponding snapshot files and delete them.
```java
try (DirectoryStream<Path> stream =
Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
for (Path path : stream) {
if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
continue;
} else {
filePatternMatches(MD5_REGEX, allMD5Files, path);
}
}
}
// first step, cleanup old snapshot and md5 file
SingleFileSnapshotInfo snapshotInfo =
cleanupOldFiles(
allSnapshotFiles,
snapshotRetentionPolicy.getNumSnapshotsRetained(),
false,
null);
// second step, cleanup only old md5 file
cleanupOldFiles(
allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
```
Have you found out why ratis will left with md5 files only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1365067081
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
> Due to the Ratis community is not very active, the old logic has been verified for a while. Can you verify this PR in your test environment?
@FMX @waitinfuture take a look again when free.
Yeah, I startup a celeborn cluster(3 master with 1 worker), and run some shuffle examples.
And let `raft.server.snapshot.retention.file.num` to `3`, after tests, obtained the expected results.
**Conf**
```
celeborn.master.ha.ratis.raft.server.storage.dir /opt/data/celeborn_ratis/
celeborn.master.ha.ratis.raft.server.snapshot.auto.trigger.enabled true
celeborn.master.ha.ratis.raft.server.snapshot.auto.trigger.threshold 1
celeborn.master.ha.ratis.raft.server.snapshot.retention.file.num 3
```
**Snapshots**
```
root@kaaa:/opt/data#
root@kaaa:/opt/data# curl 172.17.0.3:9098/masterGroupInfo
====================== Master Group INFO ==============================
group id: c5196f6d-2c34-3ed3-8b8a-47bede733167
leader info: 1(kaaa.hadoop.com:9872)
[server {
id: "2"
address: "172.17.0.3:9872"
clientAddress: "172.17.0.3:9097"
startupRole: FOLLOWER
}
commitIndex: 82
, server {
id: "1"
address: "kaaa.hadoop.com:9872"
clientAddress: "172.17.0.2:9097"
startupRole: FOLLOWER
}
commitIndex: 82
, server {
id: "3"
address: "172.17.0.4:9872"
clientAddress: "172.17.0.4:9097"
startupRole: FOLLOWER
}
commitIndex: 82
]
root@kaaa:/opt/data#
root@kaaa:/opt/data#
root@kaaa:/opt/data# ll /opt/data/celeborn_ratis/c5196f6d-2c34-3ed3-8b8a-47bede733167/sm
total 24
drwxr-xr-x 2 root root 144 Oct 19 07:50 ./
drwxr-xr-x 5 root root 61 Oct 19 07:31 ../
-rw-r--r-- 1 root root 175 Oct 19 07:50 snapshot.1_80
-rw-r--r-- 1 root root 48 Oct 19 07:50 snapshot.1_80.md5
-rw-r--r-- 1 root root 175 Oct 19 07:50 snapshot.1_82
-rw-r--r-- 1 root root 48 Oct 19 07:50 snapshot.1_82.md5
-rw-r--r-- 1 root root 175 Oct 19 07:50 snapshot.1_84
-rw-r--r-- 1 root root 48 Oct 19 07:50 snapshot.1_84.md5
root@kaaa:/opt/data#
root@kaaa:/opt/data#
root@kaaa:/opt/data#
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#issuecomment-1754263569
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1966?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1966](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1966?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (f56e439) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/56276e910f3489bc070846a19813e322b590fd17?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (56276e9) will **increase** coverage by `0.16%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1966 +/- ##
==========================================
+ Coverage 46.96% 47.11% +0.16%
==========================================
Files 164 164
Lines 10362 10362
Branches 957 957
==========================================
+ Hits 4865 4881 +16
+ Misses 5179 5166 -13
+ Partials 318 315 -3
```
[see 3 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1966/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#issuecomment-1763572598
Hi @waitinfuture, help to take a review when free, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1361760810
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
> Have you found out why ratis will left with md5 files only?
Hi @FMX, these md5 files are generated in `MasterStateMachineSuiteJ#testSnapshotCleanup` test case.
From the ratis project, we can see that clean up the md5 files if the corresponding snapshot file does not exist.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "xleoken (via GitHub)" <gi...@apache.org>.
xleoken commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1361768497
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
I add `Arrays.stream(remainingFiles).forEach(item -> System.out.println(item));` to `MasterStateMachineSuiteJ#testSnapshotCleanup`.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1030] Improve the logic of delete md5 files when initializing SimpleStateMachineStorage [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #1966:
URL: https://github.com/apache/incubator-celeborn/pull/1966#discussion_r1368059381
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/clustermeta/ha/StateMachine.java:
##########
@@ -72,94 +61,7 @@
public class StateMachine extends BaseStateMachine {
private static final Logger LOG = LoggerFactory.getLogger(StateMachine.class);
- public static final Pattern MD5_REGEX = Pattern.compile("snapshot\\.(\\d+)_(\\d+)\\.md5");
-
- private final SimpleStateMachineStorage storage =
- new SimpleStateMachineStorage() {
- /**
- * we need to delete md5 file as the same time as snapshot file deleted, so we override the
- * SimpleStateMachineStorage.cleanupOldSnapshots method, add delete md5 file action.
- *
- * @param snapshotRetentionPolicy snapshot retention policy
- * @throws IOException
- */
- @Override
- public void cleanupOldSnapshots(SnapshotRetentionPolicy snapshotRetentionPolicy)
- throws IOException {
- if (snapshotRetentionPolicy != null
- && snapshotRetentionPolicy.getNumSnapshotsRetained() > 0) {
- List<SingleFileSnapshotInfo> allSnapshotFiles = new ArrayList<>();
- List<SingleFileSnapshotInfo> allMD5Files = new ArrayList<>();
- try (DirectoryStream<Path> stream =
- Files.newDirectoryStream(SimpleStateMachineStorageUtil.getSmDir(this).toPath())) {
- for (Path path : stream) {
- if (filePatternMatches(SNAPSHOT_REGEX, allSnapshotFiles, path)) {
- continue;
- } else {
- filePatternMatches(MD5_REGEX, allMD5Files, path);
- }
- }
- }
- // first step, cleanup old snapshot and md5 file
- SingleFileSnapshotInfo snapshotInfo =
- cleanupOldFiles(
- allSnapshotFiles,
- snapshotRetentionPolicy.getNumSnapshotsRetained(),
- false,
- null);
- // second step, cleanup only old md5 file
- cleanupOldFiles(
- allMD5Files, snapshotRetentionPolicy.getNumSnapshotsRetained(), true, snapshotInfo);
- }
- }
-
- private boolean filePatternMatches(
- Pattern pattern, List<SingleFileSnapshotInfo> result, Path filePath) {
- Matcher md5Matcher = pattern.matcher(filePath.getFileName().toString());
- if (md5Matcher.matches()) {
- final long endIndex = Long.parseLong(md5Matcher.group(2));
- final long term = Long.parseLong(md5Matcher.group(1));
- final FileInfo fileInfo = new FileInfo(filePath, null);
- result.add(new SingleFileSnapshotInfo(fileInfo, term, endIndex));
- return true;
- }
- return false;
- }
-
- private SingleFileSnapshotInfo cleanupOldFiles(
- List<SingleFileSnapshotInfo> inputFiles,
- int retainedNum,
- boolean onlyCleanupMD5Files,
- SingleFileSnapshotInfo snapshotInfo) {
- SingleFileSnapshotInfo result = null;
- if (inputFiles.size() > retainedNum) {
- inputFiles.sort(new RatisSnapshotFileComparator());
- List<SingleFileSnapshotInfo> filesToBeCleaned =
- inputFiles.subList(retainedNum, inputFiles.size());
- result = filesToBeCleaned.get(0);
- for (SingleFileSnapshotInfo fileInfo : filesToBeCleaned) {
- if ((null != snapshotInfo && (fileInfo.getIndex() >= snapshotInfo.getIndex())
- || (onlyCleanupMD5Files && null == snapshotInfo))) {
- continue;
- }
- File file = fileInfo.getFile().getPath().toFile();
- if (onlyCleanupMD5Files) {
Review Comment:
@xleoken Thanks for the test, this patch will be merged into main and branch-0.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org