[GitHub] [ozone] cchenax edited a comment on pull request #3118: HDDS-6353. EC: Add padding problem

[GitBox <gi...@apache.org>]

cchenax edited a comment on pull request #3118:
URL: https://github.com/apache/ozone/pull/3118#issuecomment-1053957447


   > > maybe, but I think it should use the max length to limit the buffer.if capacity is 1048576, but the databuffer is 1050000, so the parityCellSize is 1424, so when addpadding,it limit 1424, so I think it is a problem
   > 
   > We set the buffer limits to equal the parity cell size before doing the encoding. If we have a partial stripe as the last stripe in the block, where:
   > 
   > data_1: 1000 bytes data_2: 0 data_3: 0
   > 
   > Then the parity only needs to be 1000 bytes long. Rather than pad data_1, 2 and 3 out to 1MB, we simply set all the limits to 1000, then pad data_2 and 3 with zeros to the limit of 1000. Then we generate the parity, where each of them will be 1000 long.
   > 
   > When this is written down to the datanodes, we do not store the padding. data_2 and data_3 will not get anything for this write. data_1 and parity_1 and parity_2 will get 1000 bytes.
   > 
   > Another scenario, is a small key of only 1000 bytes. It works much the same, only nothing will get written to data_2 and data_3 at all.
   
   I know what you mean
   each chunksize 1000bytes
   I mean if the data len is 1050 byte
   data_1: 1000 bytes
   data_2: 50 bytes
   data_3: 0 bytes
   so when add padding
   the data_2 and data_3 will add padding to 50bytes
   I think this is a problem, because of it not use the longest size to limit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org