You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2025/01/10 10:37:43 UTC
[PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
swamirishi opened a new pull request, #7676:
URL: https://github.com/apache/ozone/pull/7676
## What changes were proposed in this pull request?
Currently while checking file links, the exclude sst file list & files already present in the current tarball is checked in the entries by sequentially iterating through the entries, for each and every file in the om data directory (snapshot directory, active om.db, compaction backup sst file). Now if the exclude list or files present in the tarball is really long order of 1000s and the total number of sst files are in the order of millions, the bootstrap is going to read timeout and might take hours. We need to optimize and not perform this unnecessary iteration to avoid this n^2 operation and do it in O
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12064
## How was this patch tested?
Existing unit tests modification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7676:
URL: https://github.com/apache/ozone/pull/7676#discussion_r1912393778
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java:
##########
@@ -210,12 +216,12 @@ public static Map<Path, Path> normalizeExcludeList(
sstBackupDir.getOriginalDir().toString().length() + 1;
Path srcPath = Paths.get(sstBackupDir.getTmpDir().toString(),
truncateFileName(truncateLength, destPath));
- paths.put(srcPath, destPath);
+ paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(srcPath, destPath);
} else if (!s.startsWith(OM_SNAPSHOT_DIR)) {
Path fixedPath = Paths.get(checkpointLocation.toString(), s);
- paths.put(fixedPath, fixedPath);
+ paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(fixedPath, fixedPath);
} else {
- paths.put(destPath, destPath);
+ paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(destPath, destPath);
Review Comment:
Please move common `paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>())` before the `if-else` branching.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java:
##########
@@ -168,13 +169,14 @@ public void writeDbDataToStream(DBCheckpoint checkpoint,
differ.getCompactionLogDir());
// Files to be excluded from tarball
- Map<Path, Path> sstFilesToExclude = normalizeExcludeList(toExcludeList,
+ Map<String, Map<Path, Path>> sstFilesToExclude = normalizeExcludeList(toExcludeList,
checkpoint.getCheckpointLocation(), sstBackupDir);
boolean completed = getFilesForArchive(checkpoint, copyFiles,
hardLinkFiles, sstFilesToExclude, includeSnapshotData(request),
excludedList, sstBackupDir, compactionLogDir);
- writeFilesToArchive(copyFiles, hardLinkFiles, archiveOutputStream,
- completed, checkpoint.getCheckpointLocation());
+ writeFilesToArchive(copyFiles.values().stream().flatMap(map -> map.entrySet().stream())
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)),
+ hardLinkFiles, archiveOutputStream, completed, checkpoint.getCheckpointLocation());
Review Comment:
Please move stream pipeline to a separate statement, something like:
```java
Map<Path, Path> flatCopyFiles = copyFiles.values().stream()
.flatMap(map -> map.entrySet().stream())
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
writeFilesToArchive(flatCopyFiles, hardLinkFiles, archiveOutputStream,
completed, checkpoint.getCheckpointLocation());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7676:
URL: https://github.com/apache/ozone/pull/7676#discussion_r1915109255
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java:
##########
@@ -216,12 +218,12 @@ public static Map<String, Map<Path, Path>> normalizeExcludeList(
sstBackupDir.getOriginalDir().toString().length() + 1;
Path srcPath = Paths.get(sstBackupDir.getTmpDir().toString(),
truncateFileName(truncateLength, destPath));
- paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(srcPath, destPath);
+ paths.get(fileName.toString()).put(srcPath, destPath);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi merged PR #7676:
URL: https://github.com/apache/ozone/pull/7676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #7676:
URL: https://github.com/apache/ozone/pull/7676#discussion_r1914424991
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java:
##########
@@ -210,12 +216,12 @@ public static Map<Path, Path> normalizeExcludeList(
sstBackupDir.getOriginalDir().toString().length() + 1;
Path srcPath = Paths.get(sstBackupDir.getTmpDir().toString(),
truncateFileName(truncateLength, destPath));
- paths.put(srcPath, destPath);
+ paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(srcPath, destPath);
} else if (!s.startsWith(OM_SNAPSHOT_DIR)) {
Path fixedPath = Paths.get(checkpointLocation.toString(), s);
- paths.put(fixedPath, fixedPath);
+ paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(fixedPath, fixedPath);
} else {
- paths.put(destPath, destPath);
+ paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(destPath, destPath);
Review Comment:
done
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java:
##########
@@ -168,13 +169,14 @@ public void writeDbDataToStream(DBCheckpoint checkpoint,
differ.getCompactionLogDir());
// Files to be excluded from tarball
- Map<Path, Path> sstFilesToExclude = normalizeExcludeList(toExcludeList,
+ Map<String, Map<Path, Path>> sstFilesToExclude = normalizeExcludeList(toExcludeList,
checkpoint.getCheckpointLocation(), sstBackupDir);
boolean completed = getFilesForArchive(checkpoint, copyFiles,
hardLinkFiles, sstFilesToExclude, includeSnapshotData(request),
excludedList, sstBackupDir, compactionLogDir);
- writeFilesToArchive(copyFiles, hardLinkFiles, archiveOutputStream,
- completed, checkpoint.getCheckpointLocation());
+ writeFilesToArchive(copyFiles.values().stream().flatMap(map -> map.entrySet().stream())
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)),
+ hardLinkFiles, archiveOutputStream, completed, checkpoint.getCheckpointLocation());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7676:
URL: https://github.com/apache/ozone/pull/7676#issuecomment-2594583572
@swamirishi please trim extended details when merging:
```
HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links (#7676)
* HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links
Change-Id: I6871db471adc1790ac3a0ff295a4db6eeb7608ad
* HDDS-12064. Fix findbugs
Change-Id: If6f300d6068c4be2c8da99fdef3ae8495680d5ea
* HDDS-12064. Address review comments
Change-Id: Ic2b623cdb5ea6cbdcfad2b82ebb11bad62caa6d2
* HDDS-12064. Address review comments
Change-Id: I03befbcab5d08add580c44cc7ee52dbfaeb101ba
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7676:
URL: https://github.com/apache/ozone/pull/7676#discussion_r1914519250
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java:
##########
@@ -216,12 +218,12 @@ public static Map<String, Map<Path, Path>> normalizeExcludeList(
sstBackupDir.getOriginalDir().toString().length() + 1;
Path srcPath = Paths.get(sstBackupDir.getTmpDir().toString(),
truncateFileName(truncateLength, destPath));
- paths.computeIfAbsent(fileName.toString(), (k) -> new HashMap<>()).put(srcPath, destPath);
+ paths.get(fileName.toString()).put(srcPath, destPath);
Review Comment:
Please assign result of `paths.computeIfAbsent` to a variable and use it instead of `paths.get(...)`.
(Sorry if this was not clear from previous review comment.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #7676:
URL: https://github.com/apache/ozone/pull/7676#issuecomment-2593737632
@adoroszlai @prashantpogde if you don't have any issues with the patch. Can you give me a +1 for this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #7676:
URL: https://github.com/apache/ozone/pull/7676#issuecomment-2593741024
Thanks for the review on the patch @prashantpogde @adoroszlai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links [ozone]
Posted by "nandakumar131 (via GitHub)" <gi...@apache.org>.
nandakumar131 commented on PR #7676:
URL: https://github.com/apache/ozone/pull/7676#issuecomment-2599721807
> @swamirishi please trim extended details when merging:
>
> ```
> HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links (#7676)
>
> * HDDS-12064. Optimize bootstrap logic to reduce loop while checking file links
>
> Change-Id: I6871db471adc1790ac3a0ff295a4db6eeb7608ad
>
> * HDDS-12064. Fix findbugs
>
> Change-Id: If6f300d6068c4be2c8da99fdef3ae8495680d5ea
>
> * HDDS-12064. Address review comments
>
> Change-Id: Ic2b623cdb5ea6cbdcfad2b82ebb11bad62caa6d2
>
> * HDDS-12064. Address review comments
>
> Change-Id: I03befbcab5d08add580c44cc7ee52dbfaeb101ba
> ```
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org