You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "guohao-rosicky (via GitHub)" <gi...@apache.org> on 2023/09/20 09:08:02 UTC

[GitHub] [ozone] guohao-rosicky opened a new pull request, #5331: HDDS-9329. Optimize the write iops when download container

guohao-rosicky opened a new pull request, #5331:
URL: https://github.com/apache/ozone/pull/5331

   ## What changes were proposed in this pull request?
   
   Optimize the write iops when download container, add a write buffer, default use 8MB
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-9329
   
   ## How was this patch tested?
   
   existing ut
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on pull request #5331: HDDS-9329. Optimize the write iops when download container

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5331:
URL: https://github.com/apache/ozone/pull/5331#issuecomment-1729832909

   Should we close this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] guohao-rosicky commented on pull request #5331: HDDS-9329. Optimize the write iops when download container

Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5331:
URL: https://github.com/apache/ozone/pull/5331#issuecomment-1727557479

   hi, @sodonnel 
   I think you are talking about dirty_expire of the filesystem, this threshold is determined by the kernel parameter, I did a simple test and the result is as follows:
   
   **test code:**
   ```java
   public static void main(String[] args) throws IOException {
       final long filesize = 1024 * 1024 * 1024;
       final File file = new File("/Users/rosicky/test/file1.txt");
       final File file2 = new File("/Users/rosicky/test/file2.txt");
   
       final FileOutputStream fileOutputStream = new FileOutputStream(file);
       final BufferedOutputStream bufferOutputStream =
           new BufferedOutputStream(new FileOutputStream(file2), 8 * 1024 * 1024);
       final byte[] data = new byte[512];
   
       final long startTime = System.nanoTime();
       writeData(fileOutputStream, data, filesize);
       long cost1 = System.nanoTime() - startTime;
       System.out.println("file output cost: " + cost1 + "ns");
   
   
       final long startTime2 = System.nanoTime();
       writeData(bufferOutputStream, data, filesize);
       long cost2 = System.nanoTime() - startTime2;
       System.out.println("buffered output cost: " + cost2 + "ns");
     }
   
     public static void writeData(OutputStream outputStream, byte[] data,
                                  long filesize)
         throws IOException {
       for (int i = 0; i < filesize; i += data.length) {
         outputStream.write(data);
       }
       outputStream.close();
     }
   ```
   
   **result:**
   ```
   file output cost: 11646780329ns
   buffered output cost: 523351965ns
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on pull request #5331: HDDS-9329. Optimize the write iops when download container

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5331:
URL: https://github.com/apache/ozone/pull/5331#issuecomment-1727322347

   Have you performed any benchmarks to prove wrapping the stream with a buffered stream makes things faster? It is my understanding that a write to a FileoutputStream is buffered in places as the data flows to the OS and onto the disk, so this wrapping is probably not needed.
   
   Also, 8MB seems like quite a large buffer?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] guohao-rosicky closed pull request #5331: HDDS-9329. Optimize the write iops when download container

Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky closed pull request #5331: HDDS-9329. Optimize the write iops when download container
URL: https://github.com/apache/ozone/pull/5331


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] sodonnel commented on pull request #5331: HDDS-9329. Optimize the write iops when download container

Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #5331:
URL: https://github.com/apache/ozone/pull/5331#issuecomment-1727981617

   Thanks for adding some benchmark code.
   
   Testing very small writes of 512 bytes will obviously show the buffered reader as better. However varying the buffer sizes:
   
   ```
   file output cost:     6009677083ns 512b writes
   buffered output cost:  275302167ns 8MB buffer
   
   file output cost:     1365136125ns 4kb writes 
   buffered output cost:  286801417ns 8MB buffer
   
   file output cost:     564035208ns  16kb writes
   buffered output cost: 336380791ns  8MB buffer
   
   file output cost:     368352375ns 32kb writes
   buffered output cost: 290840542ns 8MB buffer
   
   file output cost:     352088292ns 32kb writes
   buffered output cost: 209192667ns 1MB buffer
   
   file output cost:     370499875ns 3kb writes
   buffered output cost: 219296750ns 0.5MB buffer
   
   file output cost:     253621584ns 32kb writes
   buffered output cost: 264730833ns 64kb buffer
   
   file output cost:     266869708ns 64kb writes
   buffered output cost: 278883500ns 64kb buffer
   ```
   
   From this it is clear that a 8MB buffer is much too large. The writes were faster with a 0.5MB buffer and in the same area for even a 64KB buffer.
   
   So the next question is - what are the size of the writes in the real download, as they are coming over the network in chunks?
   
   This change has modified GrpcReplicationClient which receives chunks of CopyContainerResponseProto. This seems to come from CopyContainerResponseStream, where we can see the constructor is passed a BUFFER_SIZE:
   
   ```
     CopyContainerResponseStream(
         CallStreamObserver<CopyContainerResponseProto> streamObserver,
         long containerId, int bufferSize) {
       super(streamObserver, containerId, bufferSize);
   }
   ```
   
   If you look inside GrpcOutputStream, you can see it fills the buffer before sending anything over the network, so that buffer size will determine the size of the chunks at the receiver. In GrpcReplicationService it defines the buffer size:
   
   ```
   static final int BUFFER_SIZE = 1024 * 1024;
   ```
   
   So its 1MB. Re-running the test before with 8MB buffer and 1MB writes:
   
   ```
   file output cost:     293118958ns
   buffered output cost: 330165959ns
   
   file output cost:     201044125ns
   buffered output cost: 360208875ns
   
   file output cost:     242833833ns
   buffered output cost: 363730708ns
   ```
   
   I ran it 3 times, and each time the change here made it slower, as well as requiring a new 8MB buffer for each stream, along with the 1MB buffer already in use, making it 9MB of overhead per concurrent download stream.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] guohao-rosicky commented on pull request #5331: HDDS-9329. Optimize the write iops when download container

Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5331:
URL: https://github.com/apache/ozone/pull/5331#issuecomment-1730674651

   Thanks @sodonnel  for the test, I think your results speak for themselves, I'm closing this pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org