You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/07/30 19:07:00 UTC
[jira] [Resolved] (HADOOP-17812) NPE in S3AInputStream read() after
failure to reconnect to store
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-17812.
-------------------------------------
Fix Version/s: 3.4.0
Resolution: Fixed
fixed in 3.4; backport to 3.3.2 planned
> NPE in S3AInputStream read() after failure to reconnect to store
> ----------------------------------------------------------------
>
> Key: HADOOP-17812
> URL: https://issues.apache.org/jira/browse/HADOOP-17812
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.2.2, 3.3.1
> Reporter: Bobby Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: failsafe-report.html.gz, s3a-test.tar.gz
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> when [reading from S3a storage|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L450], SSLException (which extends IOException) happens, which will trigger [onReadFailure|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L458].
> onReadFailure calls "reopen". it will first close the original *wrappedStream* and set *wrappedStream = null*, and then it will try to [re-get *wrappedStream*|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L184]. But what if the previous code [obtaining S3Object|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L183] throw exception, then "wrappedStream" will be null.
> And the [retry|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L446] mechanism may re-execute the [wrappedStream.read|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L450] and cause NPE.
>
> For more details, please refer to [https://github.com/NVIDIA/spark-rapids/issues/2915]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org