[GitHub] [ozone] errose28 commented on pull request #3286: HDDS-6556. Update usedBytes only when commit key.

[GitBox <gi...@apache.org>]

errose28 commented on PR #3286:
URL: https://github.com/apache/ozone/pull/3286#issuecomment-1095357010

   Hey thanks for the quick PR and discussions, sorry I am a bit late joining here. @sodonnel definitely raised some valid points. Here is how I was thinking this could work, let me know your thoughts.
   
   - Each allocate block request checks that `existing open key size` + `the block to be allocated` + `current quota usage` <= `max quota`. If this condition fails, allocate block fails.
   - Each key commit checks that `total key size` + `current quota usage` <= `max quota`. If so, it updates the quota usage and commits the key.
   
   I think this would address most of the already mentioned problems:
   - Open keys, which are owned by the system, are not counted against users' quotas.
   - If a large key write exceeds the quota, it fails as soon as possible (when the block pushing it over the quota is allocated), which might be the best we can do in a streaming scenario.
       - This also prevents large uncommitted key writes from vastly exceeding the quota.
   - Quota usage only reflects committed data.
   - Open keys that cumulatively exceed the quota can coexist independently, and only need to acknowledge each other once committed.
       - For example:
           1. 1GB quota remaining
           2. Two 1GB open keys being written: k1 and k2.
               - k1 and k2 both successfully allocate all blocks they need and write them. Neither is committed so they have no knowledge of each other.
           3. k1 is committed.
           4. k2 either calls commit, or attempts to allocate another block. This would cause it to exceed the quota that k1 just filled up, so k2's write fails at this point.
       
   The case @captainzmc mentioned where remaining quota is less than 3X block size would still be a problem with this approach. I am not sure of a good solution for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org