You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "bknbkn (via GitHub)" <gi...@apache.org> on 2024/10/14 05:37:52 UTC
[PR] [core] Support parallel reading of local orphan clean [paimon]
bknbkn opened a new pull request, #4320:
URL: https://github.com/apache/paimon/pull/4320
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
Increase parallel reading of snapshot, manifest, and data file meta to improve execution efficiency
### Tests
<!-- List UT and IT cases to verify this change -->
Test in table with 7TB size with local mode:
Before this patch: local orphan clean will cost 20 min
After this patcgh, use the same sql in parallelism 16, the procedure only cost 5 min
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1799151631
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
What about change randomlyOnlyExecute to sequentialBatchedExecute?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1802399899
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I test ```sequentialBatchedExecute``` ```randomlyOnlyExecute``` and ```randomlyExecute``` on table with 2PB in 16 parallelism, None of them happened OOM. Besides ```sequentialBatchedExecute``` procedure cost **30 min** , ```randomlyOnlyExecute``` and ```randomlyExecute``` cost about **20 min**
**So I prefer to use randomlyOnlyExecute** because this method is more concise and no other variables are introduced, it can also reduce the gc process of temporary data files variable. The disadvantage of this method is that there will be waiting lock overhead while inserting useFiles, but this has no impact on the overall time consumption, because the overall time consumption mainly comes from IO.
What do you think? @JingsongLi @wwj6591812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1798896641
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -472,13 +476,18 @@ public List<Snapshot> safelyGetAllSnapshots() throws IOException {
.map(id -> snapshotPath(id))
Review Comment:
fix
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -472,13 +476,18 @@ public List<Snapshot> safelyGetAllSnapshots() throws IOException {
.map(id -> snapshotPath(id))
.collect(Collectors.toList());
- List<Snapshot> snapshots = new ArrayList<>();
- for (Path path : paths) {
- try {
- snapshots.add(Snapshot.fromJson(fileIO.readFileUtf8(path)));
- } catch (FileNotFoundException ignored) {
- }
- }
+ List<Snapshot> snapshots = Collections.synchronizedList(new ArrayList<>());
Review Comment:
fix
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -489,18 +498,36 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
.map(id -> longLivedChangelogPath(id))
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1798856045
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -489,18 +498,36 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
.map(id -> longLivedChangelogPath(id))
Review Comment:
this::longLivedChangelogPath
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -472,13 +476,18 @@ public List<Snapshot> safelyGetAllSnapshots() throws IOException {
.map(id -> snapshotPath(id))
Review Comment:
this::snapshotPath
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -472,13 +476,18 @@ public List<Snapshot> safelyGetAllSnapshots() throws IOException {
.map(id -> snapshotPath(id))
.collect(Collectors.toList());
- List<Snapshot> snapshots = new ArrayList<>();
- for (Path path : paths) {
- try {
- snapshots.add(Snapshot.fromJson(fileIO.readFileUtf8(path)));
- } catch (FileNotFoundException ignored) {
- }
- }
+ List<Snapshot> snapshots = Collections.synchronizedList(new ArrayList<>());
Review Comment:
Collections.synchronizedList(new ArrayList<>(paths.size()));
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -489,18 +498,36 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
.map(id -> longLivedChangelogPath(id))
.collect(Collectors.toList());
- List<Changelog> changelogs = new ArrayList<>();
- for (Path path : paths) {
- try {
- String json = fileIO.readFileUtf8(path);
- changelogs.add(Changelog.fromJson(json));
- } catch (FileNotFoundException ignored) {
- }
- }
+ List<Changelog> changelogs = Collections.synchronizedList(new ArrayList<>());
+ collectSnapshots(
+ path -> {
+ try {
+ changelogs.add(Changelog.fromJson(fileIO.readFileUtf8(path)));
+ } catch (IOException e) {
+ if (!(e instanceof FileNotFoundException)) {
+ throw new RuntimeException(e);
+ }
+ }
+ },
+ paths);
return changelogs;
}
+ private void collectSnapshots(Consumer<Path> pathConsumer, List<Path> paths)
+ throws IOException {
+ ExecutorService executor =
Review Comment:
Why not use ThreadPoolUtils#createCachedThreadPool?
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I think if manifests size is very big, use randomlyOnlyExecute may oom?
Do you think so?
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -489,18 +498,36 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
.map(id -> longLivedChangelogPath(id))
.collect(Collectors.toList());
- List<Changelog> changelogs = new ArrayList<>();
- for (Path path : paths) {
- try {
- String json = fileIO.readFileUtf8(path);
- changelogs.add(Changelog.fromJson(json));
- } catch (FileNotFoundException ignored) {
- }
- }
+ List<Changelog> changelogs = Collections.synchronizedList(new ArrayList<>());
Review Comment:
Collections.synchronizedList(new ArrayList<>(paths.size()));
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1798897217
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -489,18 +498,36 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
.map(id -> longLivedChangelogPath(id))
.collect(Collectors.toList());
- List<Changelog> changelogs = new ArrayList<>();
- for (Path path : paths) {
- try {
- String json = fileIO.readFileUtf8(path);
- changelogs.add(Changelog.fromJson(json));
- } catch (FileNotFoundException ignored) {
- }
- }
+ List<Changelog> changelogs = Collections.synchronizedList(new ArrayList<>());
Review Comment:
fix
##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -489,18 +498,36 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
.map(id -> longLivedChangelogPath(id))
.collect(Collectors.toList());
- List<Changelog> changelogs = new ArrayList<>();
- for (Path path : paths) {
- try {
- String json = fileIO.readFileUtf8(path);
- changelogs.add(Changelog.fromJson(json));
- } catch (FileNotFoundException ignored) {
- }
- }
+ List<Changelog> changelogs = Collections.synchronizedList(new ArrayList<>());
+ collectSnapshots(
+ path -> {
+ try {
+ changelogs.add(Changelog.fromJson(fileIO.readFileUtf8(path)));
+ } catch (IOException e) {
+ if (!(e instanceof FileNotFoundException)) {
+ throw new RuntimeException(e);
+ }
+ }
+ },
+ paths);
return changelogs;
}
+ private void collectSnapshots(Consumer<Path> pathConsumer, List<Path> paths)
+ throws IOException {
+ ExecutorService executor =
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1799397162
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
Yeah, I repleaced it with sequentialBatchedExecute since It has better control over memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1802399899
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I test ```sequentialBatchedExecute``` ```randomlyOnlyExecute``` and ```randomlyExecute``` in **16 parallelism** on table with **2PB** of data , None of them happened OOM.
Besides ```sequentialBatchedExecute``` procedure cost **30 min** , ```randomlyOnlyExecute``` and ```randomlyExecute``` cost about **20 min**
**So I prefer to use randomlyOnlyExecute** because this method is more concise and no other variables are introduced, it can also reduce the gc process of temporary data files variable. The disadvantage of this method is that there will be waiting lock overhead while inserting useFiles, but this has no impact on the overall time consumption, because the overall time consumption mainly comes from IO.
What do you think? @JingsongLi @wwj6591812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1802399899
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I test ```sequentialBatchedExecute``` ```randomlyOnlyExecute``` and ```randomlyExecute``` on table with **2PB** in **16 parallelism**, None of them happened OOM.
Besides ```sequentialBatchedExecute``` procedure cost **30 min** , ```randomlyOnlyExecute``` and ```randomlyExecute``` cost about **20 min**
**So I prefer to use randomlyOnlyExecute** because this method is more concise and no other variables are introduced, it can also reduce the gc process of temporary data files variable. The disadvantage of this method is that there will be waiting lock overhead while inserting useFiles, but this has no impact on the overall time consumption, because the overall time consumption mainly comes from IO.
What do you think? @JingsongLi @wwj6591812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1803995528
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +118,62 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
+ Iterable<String> dataFiles =
+ sequentialBatchedExecute(
Review Comment:
replace ```sequentialBatchedExecute``` with ```randomlyOnlyExecute``` , detail in https://github.com/apache/paimon/pull/4320#discussion_r1802399899
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1800608693
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +118,62 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
+ Iterable<String> dataFiles =
+ sequentialBatchedExecute(
Review Comment:
Why using `sequentialBatchedExecute`? Maybe use `randomlyExecute`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1802399899
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I test ```sequentialBatchedExecute``` ```randomlyOnlyExecute``` and ```randomlyExecute``` in **16 parallelism** on table with **2PB** data , None of them happened OOM.
Besides ```sequentialBatchedExecute``` procedure cost **30 min** , ```randomlyOnlyExecute``` and ```randomlyExecute``` cost about **20 min**
**So I prefer to use randomlyOnlyExecute** because this method is more concise and no other variables are introduced, it can also reduce the gc process of temporary data files variable. The disadvantage of this method is that there will be waiting lock overhead while inserting useFiles, but this has no impact on the overall time consumption, because the overall time consumption mainly comes from IO.
What do you think? @JingsongLi @wwj6591812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1803995528
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +118,62 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
+ Iterable<String> dataFiles =
+ sequentialBatchedExecute(
Review Comment:
replace ```sequentialBatchedExecute``` with ```randomlyOnlyExecute```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1802399899
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I test ```sequentialBatchedExecute``` ```randomlyOnlyExecute``` and ```randomlyExecute``` on table with **2PB** in **16 parallelism**, None of them happened OOM. Besides ```sequentialBatchedExecute``` procedure cost **30 min** , ```randomlyOnlyExecute``` and ```randomlyExecute``` cost about **20 min**
**So I prefer to use randomlyOnlyExecute** because this method is more concise and no other variables are introduced, it can also reduce the gc process of temporary data files variable. The disadvantage of this method is that there will be waiting lock overhead while inserting useFiles, but this has no impact on the overall time consumption, because the overall time consumption mainly comes from IO.
What do you think? @JingsongLi @wwj6591812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4320:
URL: https://github.com/apache/paimon/pull/4320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn closed pull request #4320: [core] Support parallel reading of local orphan clean
URL: https://github.com/apache/paimon/pull/4320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support parallel reading of local orphan clean [paimon]
Posted by "bknbkn (via GitHub)" <gi...@apache.org>.
bknbkn commented on code in PR #4320:
URL: https://github.com/apache/paimon/pull/4320#discussion_r1798894639
##########
paimon-core/src/main/java/org/apache/paimon/operation/LocalOrphanFilesClean.java:
##########
@@ -113,31 +116,52 @@ public List<Path> clean() throws IOException, ExecutionException, InterruptedExc
return deleteFiles;
}
- private List<String> getUsedFiles(String branch) {
- List<String> usedFiles = new ArrayList<>();
+ private void collectWithoutDataFile(
+ String branch, Consumer<String> usedFileConsumer, Consumer<String> manifestConsumer)
+ throws IOException {
+ randomlyOnlyExecute(
+ executor,
+ snapshot -> {
+ try {
+ collectWithoutDataFile(
+ branch, snapshot, usedFileConsumer, manifestConsumer);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ safelyGetAllSnapshots(branch));
+ }
+
+ private Set<String> getUsedFiles(String branch) {
+ Set<String> usedFiles = ConcurrentHashMap.newKeySet();
ManifestFile manifestFile =
table.switchToBranch(branch).store().manifestFileFactory().create();
try {
- Set<String> manifests = new HashSet<>();
+ Set<String> manifests = ConcurrentHashMap.newKeySet();
collectWithoutDataFile(branch, usedFiles::add, manifests::add);
- List<String> dataFiles = new ArrayList<>();
- for (String manifestName : manifests) {
- retryReadingFiles(
- () -> manifestFile.readWithIOException(manifestName),
- Collections.<ManifestEntry>emptyList())
- .stream()
- .map(ManifestEntry::file)
- .forEach(
- f -> {
- if (candidateDeletes.contains(f.fileName())) {
- dataFiles.add(f.fileName());
- }
- f.extraFiles().stream()
- .filter(candidateDeletes::contains)
- .forEach(dataFiles::add);
- });
- }
- usedFiles.addAll(dataFiles);
+ randomlyOnlyExecute(
Review Comment:
I think there is indeed a risk of OOM.
But this can be avoided by reducing the parallelism (if it's equal to 1, then it's similar to serial execution before), and here it's just providing an ability to read in parallel.
If parallelism 1 is still OOM, then distributed mode is considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org