[GitHub] [ozone] sodonnel commented on pull request #3254: HDDS-5327. EC: WritableEcContainerProvider should dynamically adjust the open container groups.

[GitBox <gi...@apache.org>]

sodonnel commented on pull request #3254:
URL: https://github.com/apache/ozone/pull/3254#issuecomment-1085696777


   The formula I was suggesting is only a first guess. A lot of things can influence what the write speed will be, but the limiting factor is almost certainly the disk speed, rather than NIC speed. The NIC's will generally be much faster than the sustained disk speed.
   
   We also cannot make broad assumptions such as "ec is only for cold data" or "ec will always be large files". People will do all sort of things we cannot expect. With modern hardware, it is reasonable on HDFS to have most data be EC provided the blocks are not small, and Ozone will likely be the same.
   
   The number of open pipelines really should be a function of the write load, and the only was I can see SCM monitor that is by the rate of block requests. If the write load is very small, it doesn't matter how many pipelines are open - even one will do. We ideally want the containers to close after reaching their size limit relatively quickly. We also don't want to end up with 100 containers each with one block in them for a very slow write rate, as that could result in many small containers over time and currently we don't have a way to merge them.
   
   I guess the space tracking is one way to track the allocation rate, but have a suspicion simply tracking allocation rate would be more accurate, even though we cannot be sure how long it will take to write the given block.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org