You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Mukund Thakur (Jira)" <ji...@apache.org> on 2022/09/28 17:49:00 UTC

[jira] [Resolved] (HADOOP-18347) Restrict vectoredIO threadpool to reduce memory pressure

     [ https://issues.apache.org/jira/browse/HADOOP-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mukund Thakur resolved HADOOP-18347.
------------------------------------
    Fix Version/s: 3.3.5
       Resolution: Fixed

> Restrict vectoredIO threadpool to reduce memory pressure
> --------------------------------------------------------
>
>                 Key: HADOOP-18347
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18347
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: common, fs, fs/adl, fs/s3
>            Reporter: Rajesh Balamohan
>            Assignee: Mukund Thakur
>            Priority: Major
>              Labels: performance, pull-request-available
>             Fix For: 3.3.5
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L964-L967
> Currently, it fetches all the ranges with unbounded threadpool. This will not cause memory pressures with standard benchmarks like TPCDS. However, when large number of ranges are present with large files, this could potentially spike up memory usage of the task. Limiting the threadpool size could reduce the memory usage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org