You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sami Siren (JIRA)" <ji...@apache.org> on 2007/01/22 20:10:29 UTC
[jira] Created: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
----------------------------------------------------------
Key: HADOOP-917
URL: https://issues.apache.org/jira/browse/HADOOP-917
Project: Hadoop
Issue Type: Bug
Components: io
Affects Versions: 0.10.1
Reporter: Sami Siren
After nutch started using hadoop 0.10.1 the following Exception started to appear:
java.lang.NullPointerException
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
Anyone know the cure?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HADOOP-917:
------------------------------------
Assignee: Owen O'Malley
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Attachments: sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Sami Siren (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sami Siren updated HADOOP-917:
------------------------------
Attachment: sequencefile-npe-test.patch
Attached patch reproduces the NPE in trunk version of hadoop:
java.lang.NullPointerException
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2392)
at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2087)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:109)
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Attachments: sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-917:
---------------------------------
Fix Version/s: 0.11.1
It looks like if a map output sorter has to do a multi-way merge, it crashes with a null pointer trying to create the temporary file name.
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Fix For: 0.11.1
>
> Attachments: sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Mike Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470781 ]
Mike Smith commented on HADOOP-917:
-----------------------------------
Thanks Owen! patched fixed the problem.
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Fix For: 0.11.1
>
> Attachments: multi-level-merge.patch, sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-917:
---------------------------------
Attachment: multi-level-merge.patch
The problem was that the merge code was assuming that outputFile was set and it wasn't in that context. I've changed the API to the merge code so that the methods that don't have an output file pass in the tmpDirectory where files should be created.
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Fix For: 0.11.1
>
> Attachments: multi-level-merge.patch, sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-917:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Owen!
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Fix For: 0.11.1
>
> Attachments: multi-level-merge.patch, sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12470775 ]
Hadoop QA commented on HADOOP-917:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12350504/multi-level-merge.patch applied and successfully tested against trunk revision r503864.
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Fix For: 0.11.1
>
> Attachments: multi-level-merge.patch, sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-917) NPE in
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-917:
---------------------------------
Status: Patch Available (was: Open)
> NPE in org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue
> ----------------------------------------------------------
>
> Key: HADOOP-917
> URL: https://issues.apache.org/jira/browse/HADOOP-917
> Project: Hadoop
> Issue Type: Bug
> Components: io
> Affects Versions: 0.10.1
> Reporter: Sami Siren
> Assigned To: Owen O'Malley
> Fix For: 0.11.1
>
> Attachments: multi-level-merge.patch, sequencefile-npe-test.patch
>
>
> After nutch started using hadoop 0.10.1 the following Exception started to appear:
> java.lang.NullPointerException
> at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2158)
> at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:1892)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:498)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:191)
> at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
> Anyone know the cure?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.