[GitHub] [ozone] sodonnel commented on a change in pull request #2455: HDDS-5470 : EC: Add padding and generate parity if the last stripe is not full

[GitBox <gi...@apache.org>]

sodonnel commented on a change in pull request #2455:
URL: https://github.com/apache/ozone/pull/2455#discussion_r676774773



##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECKeyOutputStream.java
##########
@@ -473,6 +505,53 @@ public void close() throws IOException {
     ecChunkBufferCache.release();
   }
 
+  private void addPadding() {
+    final long lastStripeSize =
+        currentBlockGroupLen % (numDataBlks * ecChunkSize);
+
+    final long parityCellSize =
+        lastStripeSize < ecChunkSize ? lastStripeSize : ecChunkSize;
+    ByteBuffer[] buffers = ecChunkBufferCache.getDataBuffers();
+
+    for (int i = 0; i < numDataBlks; i++) {
+      // Pad zero bytes to make all cells exactly the size of parityCellSize
+      // If internal block is smaller than parity block, pad zero bytes.
+      // Also pad zero bytes to all parity cells
+      final int position = buffers[i].position();
+      assert position <= parityCellSize : "If an internal block is smaller"
+          + " than parity block, then its last cell should be small than last"
+          + " parity cell";
+      for (int j = 0; j < parityCellSize - position; j++) {
+        buffers[i].put((byte) 0);
+      }
+      buffers[i].flip();
+    }
+
+    buffers = ecChunkBufferCache.getParityBuffers();
+    for (int i = 0; i < (numParityBlks); i++) {
+      // Pad zero bytes to make all cells exactly the size of parityCellSize
+      // If internal block is smaller than parity block, pad zero bytes.
+      // Also pad zero bytes to all parity cells
+      final int position = buffers[i].position();
+      assert position <= parityCellSize : "If an internal block is smaller"
+          + " than parity block, then its last cell should be small than last"
+          + " parity cell";
+      for (int j = 0; j < parityCellSize - position; j++) {

Review comment:
       I don't think we need to pad the parity cells. They will always be the same length as the first data buffer.
   
   If the first data buffer is 1 byte, then we need to pad the rest of the data buffers with a single zero byte, then ensure the buffer limit is set to 1. Then we simply generate the parity and it should be a single byte. Then when writing to the stream we just write from position zero to the limit. There is no need to zero out the rest of the buffer.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org