[GitHub] [ozone] captainzmc commented on pull request #2835: HDDS-5946. [Ozone-Streaming] The 'Cannot allocate Memory 'problem occurs in client while test write

[GitBox <gi...@apache.org>]

captainzmc commented on pull request #2835:
URL: https://github.com/apache/ozone/pull/2835#issuecomment-967257731


   > I think we should not change the retry logic here. One simple way to implement buffer window size is to use a BlockingQueue; see https://issues.apache.org/jira/secure/attachment/13036034/BlockingQueue.patch . What do you think?
   
   Thanks @szetszwo for the suggest. I have looked at this patch and it is consistent with my original idea.  But this approach could create another problem. For example, my file is 128MB(smaller than the block size), chunksize is 512KB, and if streamWindowSize is 64. 
   Only 64*512KB references can be hold in this BlockingQueue.  When the file is written to chunk 65th, the entire client will be blocked and the client will be hanging. If I want the file to write successfully I have to set the streamWindowSize to 256. The client still holds 128MB of  chunk data.
   
   At present, the default block size of Ozone is 256MB, so 256MB memory needs to be reserved for each large file written, which is much larger than the original 64MB of Async write.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org