You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2020/10/22 18:52:00 UTC
[jira] [Commented] (HADOOP-13203) S3A: Support fadvise "random"
mode for high performance readPositioned() reads
[ https://issues.apache.org/jira/browse/HADOOP-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219273#comment-17219273 ]
Dongjoon Hyun commented on HADOOP-13203:
----------------------------------------
Hi, [~stevel@apache.org].
`fs.s3a.experimental.input.fadvise` seems to be experimental at 3.0.0 ~ 3.3.0.
Is there a reason to keep it as experimental? And, do we have a roadmap to make it `fs.s3a.input.fadvise` in the future releases?
> S3A: Support fadvise "random" mode for high performance readPositioned() reads
> ------------------------------------------------------------------------------
>
> Key: HADOOP-13203
> URL: https://issues.apache.org/jira/browse/HADOOP-13203
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Major
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-13203-branch-2-001.patch, HADOOP-13203-branch-2-002.patch, HADOOP-13203-branch-2-003.patch, HADOOP-13203-branch-2-004.patch, HADOOP-13203-branch-2-005.patch, HADOOP-13203-branch-2-006.patch, HADOOP-13203-branch-2-007.patch, HADOOP-13203-branch-2-008.patch, HADOOP-13203-branch-2-009.patch, HADOOP-13203-branch-2-010.patch, stream_stats.tar.gz
>
>
> Currently file's "contentLength" is set as the "requestedStreamLen", when invoking S3AInputStream::reopen(). As a part of lazySeek(), sometimes the stream had to be closed and reopened. But lots of times the stream was closed with abort() causing the internal http connection to be unusable. This incurs lots of connection establishment cost in some jobs. It would be good to set the correct value for the stream length to avoid connection aborts.
> I will post the patch once aws tests passes in my machine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org