You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/03/25 13:48:00 UTC
[GitHub] [ozone] ArafatKhan2198 opened a new pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
ArafatKhan2198 opened a new pull request #3242:
URL: https://github.com/apache/ozone/pull/3242
## What changes were proposed in this pull request?
The ContentGenerator class of the freon suite has multiple Parametrised Constructors each having various different arguments that have to be initialised, this affects the readability of the code. One way to solve this is to incorporate a builder class for ContentGenerator class so that any new parameters added in the future can be seamlessly added without the need of creating a new constructor.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6455
## How was this patch tested?
Existing UTs for ContentGeneratorClass ran successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
kaijchen commented on pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#issuecomment-1079136813
Thanks for the patch @ArafatKhan2198, I have left some comments for you to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835374121
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java
##########
@@ -48,38 +49,100 @@
private final byte[] buffer;
- ContentGenerator(long keySize, int bufferSize) {
- this(keySize, bufferSize, bufferSize);
- }
+ /**
+ * Issue Hsync after every write ( Cannot be used with Hflush ).
+ */
+ private final boolean hSync;
+
+ /**
+ * Issue Hflush after every write ( Cannot be used with Hsync ).
+ */
+ private final boolean hFlush;
- ContentGenerator(long keySize, int bufferSize, int copyBufferSize) {
- this.keySize = keySize;
- this.bufferSize = bufferSize;
- this.copyBufferSize = copyBufferSize;
+ ContentGenerator(Builder objectBuild) {
+ this.keySize = objectBuild.keySize;
+ this.bufferSize = objectBuild.bufferSize;
+ this.copyBufferSize = objectBuild.copyBufferSize;
+ this.hSync = objectBuild.hSync;
+ this.hFlush = objectBuild.hFlush;
buffer = RandomStringUtils.randomAscii(bufferSize)
.getBytes(StandardCharsets.UTF_8);
}
- /**
- * Write the required bytes to the output stream.
- */
+
public void write(OutputStream outputStream) throws IOException {
- for (long nrRemaining = keySize;
- nrRemaining > 0; nrRemaining -= bufferSize) {
+ for (long nrRemaining = keySize; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
int curSize = (int) Math.min(bufferSize, nrRemaining);
if (copyBufferSize == 1) {
for (int i = 0; i < curSize; i++) {
outputStream.write(buffer[i]);
+ flushOrSync(outputStream);
Review comment:
Does this change the code behavior?
If so, please reflect in the title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835367338
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java
##########
@@ -48,38 +49,100 @@
private final byte[] buffer;
- ContentGenerator(long keySize, int bufferSize) {
- this(keySize, bufferSize, bufferSize);
- }
+ /**
+ * Issue Hsync after every write ( Cannot be used with Hflush ).
+ */
+ private final boolean hSync;
+
+ /**
+ * Issue Hflush after every write ( Cannot be used with Hsync ).
+ */
+ private final boolean hFlush;
- ContentGenerator(long keySize, int bufferSize, int copyBufferSize) {
- this.keySize = keySize;
- this.bufferSize = bufferSize;
- this.copyBufferSize = copyBufferSize;
+ ContentGenerator(Builder objectBuild) {
+ this.keySize = objectBuild.keySize;
+ this.bufferSize = objectBuild.bufferSize;
+ this.copyBufferSize = objectBuild.copyBufferSize;
+ this.hSync = objectBuild.hSync;
+ this.hFlush = objectBuild.hFlush;
buffer = RandomStringUtils.randomAscii(bufferSize)
.getBytes(StandardCharsets.UTF_8);
}
- /**
- * Write the required bytes to the output stream.
- */
+
public void write(OutputStream outputStream) throws IOException {
- for (long nrRemaining = keySize;
- nrRemaining > 0; nrRemaining -= bufferSize) {
+ for (long nrRemaining = keySize; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
int curSize = (int) Math.min(bufferSize, nrRemaining);
if (copyBufferSize == 1) {
for (int i = 0; i < curSize; i++) {
outputStream.write(buffer[i]);
+ flushOrSync(outputStream);
}
- } else {
+ } else { /**
+ * Write the required bytes to the output stream.
+ */
for (int i = 0; i < curSize; i += copyBufferSize) {
- outputStream.write(buffer, i,
- Math.min(copyBufferSize, curSize - i));
+ outputStream.write(buffer, i, Math.min(copyBufferSize, curSize - i));
+ flushOrSync(outputStream);
}
}
}
}
+ private void flushOrSync(OutputStream outputStream) throws IOException {
+ if (outputStream instanceof FSDataOutputStream) {
+ if (hSync) {
+ ((FSDataOutputStream) outputStream).hsync();
+ } else if (hFlush) {
+ ((FSDataOutputStream) outputStream).hflush();
+ }
+ }
+ }
+
+ /**
+ * Builder Class for Content generator.
+ */
+ public static class Builder {
+
+ private long keySize = 1024;
+ private int bufferSize = 1024;
+ private int copyBufferSize = 1024;
+ private boolean hSync = false;
+ private boolean hFlush = false;
+
+ public Builder keySize(long keysize) {
Review comment:
Is it better to use `setKeySize()` as the setter name here?
Besides, after #3237 gets merged, you can use `public Builder setKeySize(long keySize)` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835535700
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java
##########
@@ -48,38 +49,100 @@
private final byte[] buffer;
- ContentGenerator(long keySize, int bufferSize) {
- this(keySize, bufferSize, bufferSize);
- }
+ /**
+ * Issue Hsync after every write ( Cannot be used with Hflush ).
+ */
+ private final boolean hSync;
+
+ /**
+ * Issue Hflush after every write ( Cannot be used with Hsync ).
+ */
+ private final boolean hFlush;
- ContentGenerator(long keySize, int bufferSize, int copyBufferSize) {
- this.keySize = keySize;
- this.bufferSize = bufferSize;
- this.copyBufferSize = copyBufferSize;
+ ContentGenerator(Builder objectBuild) {
+ this.keySize = objectBuild.keySize;
+ this.bufferSize = objectBuild.bufferSize;
+ this.copyBufferSize = objectBuild.copyBufferSize;
+ this.hSync = objectBuild.hSync;
+ this.hFlush = objectBuild.hFlush;
buffer = RandomStringUtils.randomAscii(bufferSize)
.getBytes(StandardCharsets.UTF_8);
}
- /**
- * Write the required bytes to the output stream.
- */
+
public void write(OutputStream outputStream) throws IOException {
- for (long nrRemaining = keySize;
- nrRemaining > 0; nrRemaining -= bufferSize) {
+ for (long nrRemaining = keySize; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
int curSize = (int) Math.min(bufferSize, nrRemaining);
if (copyBufferSize == 1) {
for (int i = 0; i < curSize; i++) {
outputStream.write(buffer[i]);
+ flushOrSync(outputStream);
Review comment:
When a file was closed or flushed, the file contents were written from the DataNode into the operating system using the normal close() and write() system calls.
The data may not be immediately persisted to the underlying physical storage, but may still reside in-memory in the operating system's file cache. This creates a window of vulnerability where if multiple DataNode machines fail simultaneously (e.g. loss of power to a rack), then previously written data may be lost. To combat this problem, HDFS (as of Hadoop 2.0) has introduced new APIs to provide a way to guarantee that written data will be immediately persisted to the underlying physical storage. These APIs are described in the following table.
> **hflush()** : This Method present inside the **FSDataOutputStream.java** flushes all outstanding data (i.e. the current unfinished packet) from the client into the OS buffers on all DataNode replicas.
> **hsync()** : This Method present inside the **FSDataOutputStream.java** flushes the data to the DataNodes, like hflush(), but should also force the data to underlying physical storage via fsync (or equivalent).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835376018
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java
##########
@@ -48,38 +49,100 @@
private final byte[] buffer;
- ContentGenerator(long keySize, int bufferSize) {
- this(keySize, bufferSize, bufferSize);
- }
+ /**
+ * Issue Hsync after every write ( Cannot be used with Hflush ).
+ */
+ private final boolean hSync;
+
+ /**
+ * Issue Hflush after every write ( Cannot be used with Hsync ).
+ */
+ private final boolean hFlush;
- ContentGenerator(long keySize, int bufferSize, int copyBufferSize) {
- this.keySize = keySize;
- this.bufferSize = bufferSize;
- this.copyBufferSize = copyBufferSize;
+ ContentGenerator(Builder objectBuild) {
+ this.keySize = objectBuild.keySize;
+ this.bufferSize = objectBuild.bufferSize;
+ this.copyBufferSize = objectBuild.copyBufferSize;
+ this.hSync = objectBuild.hSync;
+ this.hFlush = objectBuild.hFlush;
buffer = RandomStringUtils.randomAscii(bufferSize)
.getBytes(StandardCharsets.UTF_8);
}
- /**
- * Write the required bytes to the output stream.
- */
+
public void write(OutputStream outputStream) throws IOException {
- for (long nrRemaining = keySize;
- nrRemaining > 0; nrRemaining -= bufferSize) {
+ for (long nrRemaining = keySize; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
int curSize = (int) Math.min(bufferSize, nrRemaining);
if (copyBufferSize == 1) {
for (int i = 0; i < curSize; i++) {
outputStream.write(buffer[i]);
+ flushOrSync(outputStream);
}
- } else {
+ } else { /**
+ * Write the required bytes to the output stream.
+ */
for (int i = 0; i < curSize; i += copyBufferSize) {
- outputStream.write(buffer, i,
- Math.min(copyBufferSize, curSize - i));
+ outputStream.write(buffer, i, Math.min(copyBufferSize, curSize - i));
+ flushOrSync(outputStream);
}
}
}
}
+ private void flushOrSync(OutputStream outputStream) throws IOException {
+ if (outputStream instanceof FSDataOutputStream) {
+ if (hSync) {
+ ((FSDataOutputStream) outputStream).hsync();
+ } else if (hFlush) {
+ ((FSDataOutputStream) outputStream).hflush();
+ }
+ }
+ }
+
+ /**
+ * Builder Class for Content generator.
+ */
+ public static class Builder {
+
+ private long keySize = 1024;
+ private int bufferSize = 1024;
+ private int copyBufferSize = 1024;
+ private boolean hSync = false;
+ private boolean hFlush = false;
+
+ public Builder keySize(long keysize) {
Review comment:
Yea that'll help !! thanks for the suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835357880
##########
File path: hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestContentGenerator.java
##########
@@ -57,7 +61,10 @@ public void writeWithByteLevelWrite() throws IOException {
@Test
public void writeWithSmallBuffer() throws IOException {
- ContentGenerator generator = new ContentGenerator(1024, 1024, 10);
+ ContentGenerator generator =
+ new ContentGenerator.Builder().keySize(1024).bufferSize(1024)
+ .copyBufferSize(
+ 10).build();
Review comment:
Redundant newline here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kaijchen commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
kaijchen commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835367338
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/ContentGenerator.java
##########
@@ -48,38 +49,100 @@
private final byte[] buffer;
- ContentGenerator(long keySize, int bufferSize) {
- this(keySize, bufferSize, bufferSize);
- }
+ /**
+ * Issue Hsync after every write ( Cannot be used with Hflush ).
+ */
+ private final boolean hSync;
+
+ /**
+ * Issue Hflush after every write ( Cannot be used with Hsync ).
+ */
+ private final boolean hFlush;
- ContentGenerator(long keySize, int bufferSize, int copyBufferSize) {
- this.keySize = keySize;
- this.bufferSize = bufferSize;
- this.copyBufferSize = copyBufferSize;
+ ContentGenerator(Builder objectBuild) {
+ this.keySize = objectBuild.keySize;
+ this.bufferSize = objectBuild.bufferSize;
+ this.copyBufferSize = objectBuild.copyBufferSize;
+ this.hSync = objectBuild.hSync;
+ this.hFlush = objectBuild.hFlush;
buffer = RandomStringUtils.randomAscii(bufferSize)
.getBytes(StandardCharsets.UTF_8);
}
- /**
- * Write the required bytes to the output stream.
- */
+
public void write(OutputStream outputStream) throws IOException {
- for (long nrRemaining = keySize;
- nrRemaining > 0; nrRemaining -= bufferSize) {
+ for (long nrRemaining = keySize; nrRemaining > 0;
+ nrRemaining -= bufferSize) {
int curSize = (int) Math.min(bufferSize, nrRemaining);
if (copyBufferSize == 1) {
for (int i = 0; i < curSize; i++) {
outputStream.write(buffer[i]);
+ flushOrSync(outputStream);
}
- } else {
+ } else { /**
+ * Write the required bytes to the output stream.
+ */
for (int i = 0; i < curSize; i += copyBufferSize) {
- outputStream.write(buffer, i,
- Math.min(copyBufferSize, curSize - i));
+ outputStream.write(buffer, i, Math.min(copyBufferSize, curSize - i));
+ flushOrSync(outputStream);
}
}
}
}
+ private void flushOrSync(OutputStream outputStream) throws IOException {
+ if (outputStream instanceof FSDataOutputStream) {
+ if (hSync) {
+ ((FSDataOutputStream) outputStream).hsync();
+ } else if (hFlush) {
+ ((FSDataOutputStream) outputStream).hflush();
+ }
+ }
+ }
+
+ /**
+ * Builder Class for Content generator.
+ */
+ public static class Builder {
+
+ private long keySize = 1024;
+ private int bufferSize = 1024;
+ private int copyBufferSize = 1024;
+ private boolean hSync = false;
+ private boolean hFlush = false;
+
+ public Builder keySize(long keysize) {
Review comment:
Is it better to use `setKeySize()` as the setter name here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835396674
##########
File path: hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestContentGenerator.java
##########
@@ -57,7 +61,10 @@ public void writeWithByteLevelWrite() throws IOException {
@Test
public void writeWithSmallBuffer() throws IOException {
- ContentGenerator generator = new ContentGenerator(1024, 1024, 10);
+ ContentGenerator generator =
+ new ContentGenerator.Builder().keySize(1024).bufferSize(1024)
+ .copyBufferSize(
+ 10).build();
Review comment:
Sorry about that, Ill make the necessary changes !!
##########
File path: hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestContentGenerator.java
##########
@@ -48,7 +49,10 @@ public void writeWithSmallerBuffers() throws IOException {
@Test
public void writeWithByteLevelWrite() throws IOException {
- ContentGenerator generator = new ContentGenerator(1024, 1024, 1);
+ ContentGenerator generator =
+ new ContentGenerator.Builder().keySize(1024).bufferSize(1024)
+ .copyBufferSize(
+ 1).build();
Review comment:
Sorry about that, Ill make the necessary changes !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ArafatKhan2198 commented on a change in pull request #3242: HDDS-6455. Adding a Builder class to Freon ContentGenerator, so as to avoid Constructor Telescoping.
Posted by GitBox <gi...@apache.org>.
ArafatKhan2198 commented on a change in pull request #3242:
URL: https://github.com/apache/ozone/pull/3242#discussion_r835397464
##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/HadoopDirTreeGenerator.java
##########
@@ -154,14 +155,14 @@ private void createDir(long counter) throws Exception {
createSubDirRecursively(dir, 1, 1);
}
String message = "Successfully created directories & files. Total Dirs " +
- "Count=" + totalDirsCnt.get() + ", Total Files Count=" +
- timer.getCount();
+ "Count=" + totalDirsCnt.get() + ", Total Files Count=" +
+ timer.getCount();
print(message);
}
private void createSubDirRecursively(String parent, int depthIndex,
int spanIndex)
- throws Exception {
+ throws Exception {
Review comment:
Sorry about that, Ill make the necessary changes !!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org