You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/06/10 11:26:39 UTC
[GitHub] [ozone] adoroszlai commented on a change in pull request #2306: HDDS-5310. Enhance freon streaming generator to support multiple threads
adoroszlai commented on a change in pull request #2306:
URL: https://github.com/apache/ozone/pull/2306#discussion_r649081070
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
##########
@@ -76,47 +77,60 @@ public Void call() throws Exception {
generateBaseData();
timer = getMetrics().timer("streaming");
- setThreadNo(1);
runTests(this::copyDir);
return null;
}
- private void generateBaseData() throws IOException {
- Path sourceDir = testRoot.resolve("streaming-0");
- if (Files.exists(sourceDir)) {
- deleteDirRecursive(sourceDir);
- }
- Path subDir = sourceDir.resolve(subdir);
- Files.createDirectories(subDir);
- ContentGenerator contentGenerator = new ContentGenerator(fileSize,
- 1024);
-
- for (int i = 0; i < numberOfFiles; i++) {
- try (FileOutputStream out = new FileOutputStream(
- subDir.resolve("file-" + i).toFile())
- ) {
- contentGenerator.write(out);
+ private void generateBaseData() {
+ try {
+ Path sourceDir = threadRootDir().resolve("streaming-0");
+ final Path sourceDirParent = sourceDir.getParent();
+ if (sourceDirParent == null) {
+ throw new AssertionError("Empty parrent");
Review comment:
Doesn't `sourceDir.getParent()` resolve to the same path as `threadRootDir()`? Also, how can it be `null`?
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
##########
@@ -76,47 +77,60 @@ public Void call() throws Exception {
generateBaseData();
timer = getMetrics().timer("streaming");
- setThreadNo(1);
runTests(this::copyDir);
return null;
}
- private void generateBaseData() throws IOException {
- Path sourceDir = testRoot.resolve("streaming-0");
- if (Files.exists(sourceDir)) {
- deleteDirRecursive(sourceDir);
- }
- Path subDir = sourceDir.resolve(subdir);
- Files.createDirectories(subDir);
- ContentGenerator contentGenerator = new ContentGenerator(fileSize,
- 1024);
-
- for (int i = 0; i < numberOfFiles; i++) {
- try (FileOutputStream out = new FileOutputStream(
- subDir.resolve("file-" + i).toFile())
- ) {
- contentGenerator.write(out);
+ private void generateBaseData() {
+ try {
+ Path sourceDir = threadRootDir().resolve("streaming-0");
+ final Path sourceDirParent = sourceDir.getParent();
+ if (sourceDirParent == null) {
+ throw new AssertionError("Empty parrent");
}
+ if (Files.exists(sourceDirParent)) {
+ deleteDirRecursive(sourceDirParent);
+ }
+ Path subDir = sourceDir.resolve(subdir);
+ Files.createDirectories(subDir);
+ ContentGenerator contentGenerator = new ContentGenerator(fileSize,
+ 1024);
+
+ for (int i = 0; i < numberOfFiles; i++) {
+ try (FileOutputStream out = new FileOutputStream(
+ subDir.resolve("file-" + i).toFile())
+ ) {
+ contentGenerator.write(out);
+ }
+ }
+ } catch (Exception ex) {
+ throw new RuntimeException(ex);
}
}
private void copyDir(long l) {
- Path sourceDir = testRoot.resolve("streaming-" + l);
- Path destinationDir = testRoot.resolve("streaming-" + (l + 1));
+ Integer index = counter.get();
+ if (index == null) {
+ generateBaseData();
+ index = 0;
+ }
+ Path sourceDir = threadRootDir().resolve("streaming-" + index);
+ Path destinationDir = threadRootDir().resolve("streaming-" + (index + 1));
+ counter.set(index + 1);
+ int port = (int) (1234 + l);
Review comment:
Extreme number of tests may lead to invalid ports. Can it be guarded with some kind of modulo arithmetic?
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
##########
@@ -76,47 +77,60 @@ public Void call() throws Exception {
generateBaseData();
Review comment:
I guess we no longer need to generate base data in main thread.
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
##########
@@ -76,47 +77,60 @@ public Void call() throws Exception {
generateBaseData();
timer = getMetrics().timer("streaming");
- setThreadNo(1);
runTests(this::copyDir);
return null;
}
- private void generateBaseData() throws IOException {
- Path sourceDir = testRoot.resolve("streaming-0");
- if (Files.exists(sourceDir)) {
- deleteDirRecursive(sourceDir);
- }
- Path subDir = sourceDir.resolve(subdir);
- Files.createDirectories(subDir);
- ContentGenerator contentGenerator = new ContentGenerator(fileSize,
- 1024);
-
- for (int i = 0; i < numberOfFiles; i++) {
- try (FileOutputStream out = new FileOutputStream(
- subDir.resolve("file-" + i).toFile())
- ) {
- contentGenerator.write(out);
+ private void generateBaseData() {
+ try {
+ Path sourceDir = threadRootDir().resolve("streaming-0");
+ final Path sourceDirParent = sourceDir.getParent();
+ if (sourceDirParent == null) {
+ throw new AssertionError("Empty parrent");
}
+ if (Files.exists(sourceDirParent)) {
+ deleteDirRecursive(sourceDirParent);
+ }
+ Path subDir = sourceDir.resolve(subdir);
Review comment:
It's a bit confusing to have two such similar variables (`subdir` vs `subDir`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org