You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/01/17 20:57:30 UTC
[jira] Created: (HADOOP-898) namenode generates infinite stream of
null pointers
namenode generates infinite stream of null pointers
---------------------------------------------------
Key: HADOOP-898
URL: https://issues.apache.org/jira/browse/HADOOP-898
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: Owen O'Malley
Assigned To: Konstantin Shvachko
My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 50000 call error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1
621)
at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:15
63)
at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:589)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-898) namenode generates infinite stream of
null pointers
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-898:
-------------------------------
Attachment: 898-post-803-revert.patch
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: 898-post-803-revert.patch, HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465555 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
storedBlock is null.
This NPE implies node N is listed as one of the nodes that has a block B (in blocksMap) but N.blocks map does not contain B.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465634 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
Both patches worked in Owen's cluster.
Doug, could you apply HADOOP-898-2.patch to the current trunk while I look more into the problem? The first patch is much more conservative and reverts whole of HADOOP-803, we could do this if we don't find more about the problem soon. I am fairly confident of the second patch.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465824 ]
Devaraj Das commented on HADOOP-898:
------------------------------------
During a run of the sort benchmark many reduces failed with these exceptions:
EXCEPTION TRACE #1
----------------------------------
java.lang.NullPointerException
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeBackupStream(DFSClient.java:972)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1219)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.flush(DFSClient.java:1181)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.write(DFSClient.java:1163)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.write(FSDataOutputStream.java:85)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:114)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at java.io.FilterOutputStream.close(FilterOutputStream.java:140)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:558)
at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:331)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
EXCEPTION TRACE #2
---------------------------------
java.io.IOException: Trying to write to backupStream but it already closed or not open
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.flushData(DFSClient.java:1195)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.flush(DFSClient.java:1183)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1315)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:99)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:558)
at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:331)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
EXCEPTION TRACE #3 (this is Hadoop-758 I think)
----------------------------------------------------------------------
java.io.FileNotFoundException: /export/crawlspace/kryptonite/ddas/dfs/data/tmp/client-1257934407504380214 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:106)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:1229)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:1318)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:98)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:558)
at org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:72)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:331)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
EXCEPTION TRACE #4 (although this I think has something to do with RPC timing out at the namenode, I hadn't encountered this exception in a long long time)
--------------------------------------------------------
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:467)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:164)
at org.apache.hadoop.dfs.$Proxy1.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:248)
at org.apache.hadoop.dfs.DFSClient.(DFSClient.java:105)
at org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:65)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:160)
at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:91)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1366)
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-898) namenode generates infinite stream of
null pointers
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-898:
---------------------------------
Description:
My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 50000 call error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:589)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
was:
My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 50000 call error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1
621)
at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:15
63)
at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:589)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Konstantin Shvachko
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (HADOOP-898) namenode generates infinite stream of null pointers
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Rather than apply a patch that effectively reverts HADOOP-803, it
would be better to revert the patch using svn and then apply the
neccessary (one line) patch to re-introduce the change needed by the
overlapping HADOOP-855 patch. The svn command that I've applied and
tested is:
svn merge -r 496845:496844 .
(where . is my trunk directory)
I will also attach the one line diff that was necessary to apply
after the above merge command.
On Jan 17, 2007, at 11:10 PM, Nigel Daley wrote:
> Yes, the test started failing 20% of the time since HADOOP-803 was
> checked in.
>
> On Jan 17, 2007, at 11:00 PM, Raghu Angadi (JIRA) wrote:
>
>>
>> [ https://issues.apache.org/jira/browse/HADOOP-898?
>> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
>> tabpanel#action_12465664 ]
>>
>> Raghu Angadi commented on HADOOP-898:
>> -------------------------------------
>>
>>
>> Thanks Nigel. Was the test failing 20% of the time before any of
>> these patches? Was it occationally failing before HADOOP-803 was
>> checked in?
>>
>>
>>> namenode generates infinite stream of null pointers
>>> ---------------------------------------------------
>>>
>>> Key: HADOOP-898
>>> URL: https://issues.apache.org/jira/browse/
>>> HADOOP-898
>>> Project: Hadoop
>>> Issue Type: Bug
>>> Components: dfs
>>> Reporter: Owen O'Malley
>>> Assigned To: Raghu Angadi
>>> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>>>
>>>
>>> My namenode is generating a constant stream of
>>> NullPointerExceptions. The log looks like:
>>> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC
>>> Server handler 3
>>> on 50000 call error: java.io.IOException:
>>> java.lang.NullPointerException
>>> java.io.IOException: java.lang.NullPointerException
>>> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock
>>> (FSNamesystem.java:1621)
>>> at org.apache.hadoop.dfs.FSNamesystem.processReport
>>> (FSNamesystem.java:1563)
>>> at org.apache.hadoop.dfs.NameNode.blockReport
>>> (NameNode.java:573)
>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
>>> Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>>> (DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:589)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the
>> administrators: https://issues.apache.org/jira/secure/
>> Administrators.jspa
>> -
>> For more information on JIRA, see: http://www.atlassian.com/
>> software/jira
>>
>>
>
Re: [jira] Commented: (HADOOP-898) namenode generates infinite stream of null pointers
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Yes, the test started failing 20% of the time since HADOOP-803 was
checked in.
On Jan 17, 2007, at 11:00 PM, Raghu Angadi (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HADOOP-898?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12465664 ]
>
> Raghu Angadi commented on HADOOP-898:
> -------------------------------------
>
>
> Thanks Nigel. Was the test failing 20% of the time before any of
> these patches? Was it occationally failing before HADOOP-803 was
> checked in?
>
>
>> namenode generates infinite stream of null pointers
>> ---------------------------------------------------
>>
>> Key: HADOOP-898
>> URL: https://issues.apache.org/jira/browse/HADOOP-898
>> Project: Hadoop
>> Issue Type: Bug
>> Components: dfs
>> Reporter: Owen O'Malley
>> Assigned To: Raghu Angadi
>> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>>
>>
>> My namenode is generating a constant stream of
>> NullPointerExceptions. The log looks like:
>> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC
>> Server handler 3
>> on 50000 call error: java.io.IOException:
>> java.lang.NullPointerException
>> java.io.IOException: java.lang.NullPointerException
>> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock
>> (FSNamesystem.java:1621)
>> at org.apache.hadoop.dfs.FSNamesystem.processReport
>> (FSNamesystem.java:1563)
>> at org.apache.hadoop.dfs.NameNode.blockReport
>> (NameNode.java:573)
>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown
>> Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:589)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the
> administrators: https://issues.apache.org/jira/secure/
> Administrators.jspa
> -
> For more information on JIRA, see: http://www.atlassian.com/
> software/jira
>
>
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465664 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
Thanks Nigel. Was the test failing 20% of the time before any of these patches? Was it occationally failing before HADOOP-803 was checked in?
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465834 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
> Impossible situation: could not find target position 8192
> java.io.IOException: Impossible situation: could not find target position 8192
I don't think this is related. We should file a different bug.
>java.lang.NullPointerException
>at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeBackupStream(DFSClient.java:972)
Not related to this bug but caused by one of my recent patches. will fix today.
#2 is not a bug. #2 and #3 need patch in HADOOP-758.
I don't know much about #4 but my understanding is that RPC timeouts are expected with a busy nameserver.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465823 ]
Devaraj Das commented on HADOOP-898:
------------------------------------
Not sure whether the issues I see are related to this bug or not.
Along with sporadic failures of TestSmallBlock testcase, I also see TestPread failing once in a while with the exception:
Impossible situation: could not find target position 8192
java.io.IOException: Impossible situation: could not find target position 8192
at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:873)
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:61)
at org.apache.hadoop.fs.FSDataInputStream$Checker.readFully(FSDataInputStream.java:155)
at org.apache.hadoop.fs.FSDataInputStream$PositionCache.readFully(FSDataInputStream.java:212)
at org.apache.hadoop.fs.FSDataInputStream$Buffer.readFully(FSDataInputStream.java:265)
at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:315)
at org.apache.hadoop.dfs.TestPread.pReadFile(TestPread.java:92)
at org.apache.hadoop.dfs.TestPread.testPreadDFS(TestPread.java:119)
Once I also saw the TestDFSShell failing (forgot to note the exception).
The DFS has become somewhat flaky lately.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-898) namenode generates infinite stream of
null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-898:
--------------------------------
Attachment: HADOOP-898.patch
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-898) namenode generates infinite stream of
null pointers
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting resolved HADOOP-898.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.11.0
I just committed this. Thanks, Nigel & Raghu!
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Fix For: 0.11.0
>
> Attachments: 898-post-803-revert.patch, HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465606 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
That can explain this case.. I still need to get used thinking in exceptions. I will check if client got an IPC exception.
Both addStoredBlock() and addBlock() can be called outside the while() loop. So they could be just one function.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465661 ]
Nigel Daley commented on HADOOP-898:
------------------------------------
-1 for HADOOP-898-2.patch
+1 for HADOOP-898.patch.
Of the two patches, I find that only HADOOP-898.patch reduces the probability of TestSmallBlock failing from around 20% to 0% after 100+ runs.
For those interested, when TestSmallBlock fails, it's because one call to blocks[i].getNumBytes() in the FSNamesystem.getDatanodeHints() method returns 0 when it should return 1. Since it is sporadic, it seems like a synchronization/timing issue.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465534 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
This must be my changes that got checked in yesterday. I am looking into this.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Konstantin Shvachko
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465615 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
Attached much smaller 2.patch that basically check if storedBlock is null.
An IPC exception that Konstantin mentioned would create same situation before HADOOP-803 as well (i.e. containingNodes has N but N does not have the block) . I am not sure yet whether that is ok or if it could lead to some other problems. Only difference now is that we are requiring that such a situation not exist. The attached patch allows this apparent inconsistency just like before.
Both patches pass 'ant test'.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-898) namenode generates infinite stream of
null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi reassigned HADOOP-898:
-----------------------------------
Assignee: Raghu Angadi (was: Konstantin Shvachko)
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465602 ]
Konstantin Shvachko commented on HADOOP-898:
--------------------------------------------
You add block to containingNodes inside addStoredBlock(), but add this same block to the
DatanodeDescriptor.blocks outside of addStoredBlock().
If something goes wrong in addStoredBlock() and it throws an exception AFTER the block
was added to containingNodes, will that be the case we are seeing now?
We should think how to add blocks to the DatanodeDescriptor inside addStoredBlock().
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465839 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
Regd 898-post-803-revert.patch Nigel Daley wrote:
> Rather than apply a patch that effectively reverts HADOOP-803, it would
> be better to revert the patch using svn and then apply the neccessary
> (one line) patch to re-introduce the change needed by the overlapping
> HADOOP-855 patch. The svn command that I've applied and tested is:
>
> svn merge -r 496845:496844 .
>
> (where . is my trunk directory)
>
> I will also attach the one line diff that was necessary to apply after
> the above merge command.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: 898-post-803-revert.patch, HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-898) namenode generates infinite stream of
null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-898:
--------------------------------
Attachment: HADOOP-898-2.patch
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898-2.patch, HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-898) namenode generates infinite stream
of null pointers
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12465569 ]
Raghu Angadi commented on HADOOP-898:
-------------------------------------
Attached HADOOP-898.patch that reverses the patch in HADOOP-803 and one line change for patch in HADOOP-855 (changing List<> to Collection).
Owen could you try the patch in your build? Thanks.
> namenode generates infinite stream of null pointers
> ---------------------------------------------------
>
> Key: HADOOP-898
> URL: https://issues.apache.org/jira/browse/HADOOP-898
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Owen O'Malley
> Assigned To: Raghu Angadi
> Attachments: HADOOP-898.patch
>
>
> My namenode is generating a constant stream of NullPointerExceptions. The log looks like:
> 2007-01-17 19:51:27,461 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
> on 50000 call error: java.io.IOException: java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.dfs.FSNamesystem.addStoredBlock(FSNamesystem.java:1621)
> at org.apache.hadoop.dfs.FSNamesystem.processReport(FSNamesystem.java:1563)
> at org.apache.hadoop.dfs.NameNode.blockReport(NameNode.java:573)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:589)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira