You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Matthew Paduano (JIRA)" <ji...@apache.org> on 2015/12/29 02:57:49 UTC
[jira] [Commented] (HADOOP-10542) Potential null pointer
dereference in Jets3tFileSystemStore#retrieveBlock()
[ https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073363#comment-15073363 ]
Matthew Paduano commented on HADOOP-10542:
------------------------------------------
This change seems to break some things. In particular, have a closer look at:
S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException but instead IOException)
FileSystem.exists() (which no longer returns false but instead raises IOException)
S3FileSystem.create() (which no longer succeeds but instead raises IOException)
While the javadoc suggests that the API permits one to raise IOException, most of the code I have
encountered while debugging a command like "hadoop distcp hdfs://localhost:9000/test s3://xxx:yyy@com.bar.foo/"
seems to expect (1) retrieveINode() to return null and (2) FileNotFoundException to be raised when a file is not
found (i.e. when get() fails!).
2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1449826461866_0005_m_000006_0 - exited : java.io.IOException: /test doesn't exist
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170)
at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source)
at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
changing the "raise IOE..." to "return null" fixes all of the above code sites and
allows distcp to succeed.
> Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
> ---------------------------------------------------------------------------
>
> Key: HADOOP-10542
> URL: https://issues.apache.org/jira/browse/HADOOP-10542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.6.0
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Minor
> Fix For: 2.7.0
>
> Attachments: hadoop-10542-001.patch
>
>
> {code}
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> {code}
> get() may return null.
> The while loop dereferences in without null check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)