You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Mukund Thakur (Jira)" <ji...@apache.org> on 2022/06/20 22:22:00 UTC

[jira] [Resolved] (HADOOP-18106) Handle memory fragmentation in S3 Vectored IO implementation.

     [ https://issues.apache.org/jira/browse/HADOOP-18106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mukund Thakur resolved HADOOP-18106.
------------------------------------
    Resolution: Fixed

> Handle memory fragmentation in S3 Vectored IO implementation.
> -------------------------------------------------------------
>
>                 Key: HADOOP-18106
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18106
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Mukund Thakur
>            Assignee: Mukund Thakur
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> As we have implemented merging of ranges in the S3AInputStream implementation of vectored IO api, it can lead to memory fragmentation. Let me explain by example.
>  
> Suppose client requests for 3 ranges. 
> 0-500, 700-1000 and 1200-1500.
> Now because of merging, all the above ranges will get merged into one and we will allocate a big byte buffer of 0-1500 size but return sliced byte buffers for the desired ranges.
> Now once the client is done reading all the ranges, it will only be able to free the memory for requested ranges and memory of the gaps will never be released for eg here (500-700 and 1000-1200).
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org