You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Henry Robinson (Created) (JIRA)" <ji...@apache.org> on 2012/03/09 00:09:57 UTC
[jira] [Created] (HDFS-3067) Null pointer in
DFSInputStream.readBuffer if read is repeated on singly-replicated
corrupted block
Null pointer in DFSInputStream.readBuffer if read is repeated on singly-replicated corrupted block
--------------------------------------------------------------------------------------------------
Key: HDFS-3067
URL: https://issues.apache.org/jira/browse/HDFS-3067
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Henry Robinson
Assignee: Henry Robinson
With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
Here's the body of a test that reproduces the problem:
{code}
final short REPL_FACTOR = 1;
final long FILE_LENGTH = 512L;
cluster.waitActive();
FileSystem fs = cluster.getFileSystem();
Path path = new Path("/corrupted");
DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
InetSocketAddress nnAddr =
new InetSocketAddress("localhost", cluster.getNameNodePort());
DFSClient client = new DFSClient(nnAddr, conf);
DFSInputStream dis = client.open(path.toString());
byte[] arr = new byte[(int)FILE_LENGTH];
boolean sawException = false;
try {
dis.read(arr, 0, (int)FILE_LENGTH);
} catch (ChecksumException ex) {
sawException = true;
}
assertTrue(sawException);
sawException = false;
try {
dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
} catch (ChecksumException ex) {
sawException = true;
}
{code}
The stack:
{code}
java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
[snip test stack]
{code}
and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410786#comment-13410786 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2508 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2508/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230525#comment-13230525 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1954 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1954/])
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on corrupted block. Contributed by Henry Robinson. (Revision 1301182)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301182
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.24.0
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229980#comment-13229980 ]
Hadoop QA commented on HDFS-3067:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517987/HDFS-3067.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2008//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2008//console
This message is automatically generated.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Henry Robinson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated HDFS-3067:
---------------------------------
Attachment: HDFS-3067.1.patch
Patch addresses review comments.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Henry Robinson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230304#comment-13230304 ]
Henry Robinson commented on HDFS-3067:
--------------------------------------
There are two test failures:
* TestHDFSCli looks like an inherited failure - it's been failing in other pre-commit builds.
* TestDatanodeBlockScanner passes every time for me locally. The test result makes it look like the standard corrupt-a-block mechanism failed by hitting a timeout. Could this be environmental?
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230509#comment-13230509 ]
Hadoop QA commented on HDFS-3067:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517987/HDFS-3067.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.cli.TestHDFSCLI
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2010//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2010//console
This message is automatically generated.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410261#comment-13410261 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1099 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1099/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410352#comment-13410352 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1132 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1132/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3067:
---------------------------------
Target Version/s: 0.24.0
Affects Version/s: 0.24.0
Status: Patch Available (was: Open)
Marking PA for Henry so test-patch runs.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229898#comment-13229898 ]
Aaron T. Myers commented on HDFS-3067:
--------------------------------------
Just marking changing open/patch-available isn't sufficient to kick test-patch. You actually need to upload a new patch. But, I've just kicked Jenkins for you manually. Build here: https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-HDFS-Build/2008/
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230526#comment-13230526 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #1879 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1879/])
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on corrupted block. Contributed by Henry Robinson. (Revision 1301182)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301182
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.24.0
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3067:
---------------------------------
Fix Version/s: (was: 3.0.0)
2.0.1-alpha
I've just merged this patch to branch-2 and updated CHANGES.txt in trunk to suit.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230514#comment-13230514 ]
Aaron T. Myers commented on HDFS-3067:
--------------------------------------
Looks to me like the TestDatanodeBlockScanner failure was indeed unrelated.
+1, the latest patch looks good to me. I'm going to commit this momentarily.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410161#comment-13410161 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2436 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2436/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230419#comment-13230419 ]
Aaron T. Myers commented on HDFS-3067:
--------------------------------------
I bet the test failure of TestDatanodeBlockScanner is simply HDFS-2881. I've just kicked Jenkins for this patch again to see if we can get a clean run. I agree that the TestHDFSCli failure is unrelated.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3067.
-------------------------------
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Henry Robinson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated HDFS-3067:
---------------------------------
Status: Patch Available (was: Open)
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HDFS-3067:
------------------------------
Fix Version/s: 0.23.3
I've committed to branch-23.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.23.3, 2.1.0-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230561#comment-13230561 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1888 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1888/])
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on corrupted block. Contributed by Henry Robinson. (Revision 1301182)
Result = ABORTED
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301182
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.24.0
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) Null pointer in
DFSInputStream.readBuffer if read is repeated on singly-replicated
corrupted block
Posted by "Henry Robinson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated HDFS-3067:
---------------------------------
Attachment: HDFS-3607.patch
Patch + test.
The problem was that the failing first read leaves currentNode == null when it can't find any non-corrupt replicas. However the next read doesn't check to see if currentNode == null but instead only checks to see if the current position is valid when deciding whether to open another block reader and update currentNode. Adding this condition fixes the bug.
> Null pointer in DFSInputStream.readBuffer if read is repeated on singly-replicated corrupted block
> --------------------------------------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3067:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this to trunk. Thanks a lot for the contribution, Hank.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.24.0
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Henry Robinson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated HDFS-3067:
---------------------------------
Status: Open (was: Patch Available)
Kicking Hudson, which seems to have forgotten about me :(
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3067:
---------------------------------
Component/s: hdfs client
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410157#comment-13410157 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2504 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2504/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231127#comment-13231127 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Hdfs-trunk #986 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/986/])
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on corrupted block. Contributed by Henry Robinson. (Revision 1301182)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301182
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.24.0
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226724#comment-13226724 ]
Aaron T. Myers commented on HDFS-3067:
--------------------------------------
Looks pretty good to me, Hank. Just a few small nits. +1 once these are addressed.
# A few lines are over 80 chars.
# Indent 4 lines on lines that go over 80 chars, instead of 2.
# Rather than use the "sawException" boolean, add an explicit call to "fail()" after the dis.read(), and call GenericTestUtils.assertExceptionContains(...) in the catch clause.
# Put some white space around "=" and "<" in the for loop.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225787#comment-13225787 ]
Hadoop QA commented on HDFS-3067:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517646/HDFS-3607.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1975//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1975//console
This message is automatically generated.
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231197#comment-13231197 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1021 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1021/])
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on corrupted block. Contributed by Henry Robinson. (Revision 1301182)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301182
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.24.0
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Henry Robinson (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henry Robinson updated HDFS-3067:
---------------------------------
Summary: NPE in DFSInputStream.readBuffer if read is repeated on corrupted block (was: Null pointer in DFSInputStream.readBuffer if read is repeated on singly-replicated corrupted block)
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Attachments: HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416148#comment-13416148 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #316 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/316/])
svn merge -c 1301182 FIXES: HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on corrupted block. Contributed by Henry Robinson. (Revision 1362259)
Result = SUCCESS
daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1362259
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 0.23.3, 2.1.0-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410202#comment-13410202 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2454 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2454/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3067) NPE in DFSInputStream.readBuffer if
read is repeated on corrupted block
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410828#comment-13410828 ]
Hudson commented on HDFS-3067:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2441 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2441/])
Move CHANGES.txt entry for HDFS-3067 to branch-2 instead of trunk. (Revision 1359331)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1359331
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
> -----------------------------------------------------------------------
>
> Key: HDFS-3067
> URL: https://issues.apache.org/jira/browse/HDFS-3067
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.24.0
> Reporter: Henry Robinson
> Assignee: Henry Robinson
> Fix For: 2.0.1-alpha
>
> Attachments: HDFS-3067.1.patch, HDFS-3607.patch
>
>
> With a singly-replicated block that's corrupted, issuing a read against it twice in succession (e.g. if ChecksumException is caught by the client) gives a NullPointerException.
> Here's the body of a test that reproduces the problem:
> {code}
> final short REPL_FACTOR = 1;
> final long FILE_LENGTH = 512L;
> cluster.waitActive();
> FileSystem fs = cluster.getFileSystem();
> Path path = new Path("/corrupted");
> DFSTestUtil.createFile(fs, path, FILE_LENGTH, REPL_FACTOR, 12345L);
> DFSTestUtil.waitReplication(fs, path, REPL_FACTOR);
> ExtendedBlock block = DFSTestUtil.getFirstBlock(fs, path);
> int blockFilesCorrupted = cluster.corruptBlockOnDataNodes(block);
> assertEquals("All replicas not corrupted", REPL_FACTOR, blockFilesCorrupted);
> InetSocketAddress nnAddr =
> new InetSocketAddress("localhost", cluster.getNameNodePort());
> DFSClient client = new DFSClient(nnAddr, conf);
> DFSInputStream dis = client.open(path.toString());
> byte[] arr = new byte[(int)FILE_LENGTH];
> boolean sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH);
> } catch (ChecksumException ex) {
> sawException = true;
> }
>
> assertTrue(sawException);
> sawException = false;
> try {
> dis.read(arr, 0, (int)FILE_LENGTH); // <-- NPE thrown here
> } catch (ChecksumException ex) {
> sawException = true;
> }
> {code}
> The stack:
> {code}
> java.lang.NullPointerException
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:492)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:545)
> [snip test stack]
> {code}
> and the problem is that currentNode is null. It's left at null after the first read, which fails, and then is never refreshed because the condition in read that protects blockSeekTo is only triggered if the current position is outside the block's range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira