You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/07/19 19:06:00 UTC

[jira] [Commented] (HADOOP-18347) Restrict vectoredIO threadpool to reduce memory pressure

    [ https://issues.apache.org/jira/browse/HADOOP-18347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568690#comment-17568690 ] 

Steve Loughran commented on HADOOP-18347:
-----------------------------------------

makes snese. that bounded pool is fairly bounded across an fs instance, so could become a bottleneck. time to review the defaults?

> Restrict vectoredIO threadpool to reduce memory pressure
> --------------------------------------------------------
>
>                 Key: HADOOP-18347
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18347
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: common, fs, fs/adl, fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Major
>              Labels: performance
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L964-L967
> Currently, it fetches all the ranges with unbounded threadpool. This will not cause memory pressures with standard benchmarks like TPCDS. However, when large number of ranges are present with large files, this could potentially spike up memory usage of the task. Limiting the threadpool size could reduce the memory usage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org