You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/07/22 09:26:00 UTC
[jira] [Commented] (HADOOP-17812) NPE when reading s3a
[ https://issues.apache.org/jira/browse/HADOOP-17812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385373#comment-17385373 ]
Steve Loughran commented on HADOOP-17812:
-----------------------------------------
stack
{code}
17:22:13 Caused by: java.lang.NullPointerException
17:22:13 at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$read$3(S3AInputStream.java:450)
17:22:13 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
17:22:13 at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
17:22:13 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
17:22:13 at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
17:22:13 at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
17:22:13 at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:446)
17:22:13 at java.io.DataInputStream.readFully(DataInputStream.java:195)
17:22:13 at com.nvidia.spark.rapids.ParquetPartitionReaderBase.copyDataRange(GpuParquetScan.scala:509)
17:22:13 at com.nvidia.spark.rapids.ParquetPartitionReaderBase.copyDataRange$(GpuParquetScan.scala:497)
17:22:13 at com.nvidia.spark.rapids.MultiFileCloudParquetPartitionReader.copyDataRange(GpuParquetScan.scala:975)
17:22:13 at com.nvidia.spark.rapids.ParquetPartitionReaderBase.$anonfun$copyBlocksData$3(GpuParquetScan.scala:580)
17:22:13 at com.nvidia.spark.rapids.ParquetPartitionReaderBase.$anonfun$copyBlocksData$3$adapted(GpuParquetScan.scala:580)
17:22:13 at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
{code}
> NPE when reading s3a
> --------------------
>
> Key: HADOOP-17812
> URL: https://issues.apache.org/jira/browse/HADOOP-17812
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Reporter: Bobby Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> when [reading from S3a storage|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L450], SSLException (which extends IOException) happens, which will trigger [onReadFailure|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L458].
> onReadFailure calls "reopen". it will first close the original *wrappedStream* and set *wrappedStream = null*, and then it will try to [re-get *wrappedStream*|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L184]. But what if the previous code [obtaining S3Object|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L183] throw exception, then "wrappedStream" will be null.
> And the [retry|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L446] mechanism may re-execute the [wrappedStream.read|https://github.com/apache/hadoop/blob/rel/release-3.2.0/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L450] and cause NPE.
>
> For more details, please refer to [https://github.com/NVIDIA/spark-rapids/issues/2915]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org