You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jothi Padmanabhan (JIRA)" <ji...@apache.org> on 2009/09/17 12:19:57 UTC
[jira] Created: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
JobHistory should handle cases where task completion events are generated after job completion event
----------------------------------------------------------------------------------------------------
Key: MAPREDUCE-995
URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jothi Padmanabhan
Assignee: Jothi Padmanabhan
It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-995:
------------------------------------
Status: Open (was: Patch Available)
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: M995-2.patch, mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-995:
------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I committed this. Thanks, Jothi!
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: M995-2.patch, mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-995:
------------------------------------
Attachment: M995-2.patch
This should use a Collections.synchronizedMap wrapper rather than a ConcurrentHashMap, since it doesn't actually use the atomic get/put operations, but only requires mutual exclusion.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: M995-2.patch, mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756558#action_12756558 ]
Hadoop QA commented on MAPREDUCE-995:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419875/mapred-995-v1.patch
against trunk revision 816147.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/97/console
This message is automatically generated.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-995:
----------------------------------------
Status: Open (was: Patch Available)
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-995:
----------------------------------------
Status: Patch Available (was: Open)
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-995:
----------------------------------------
Status: Patch Available (was: Open)
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756466#action_12756466 ]
Hong Tang commented on MAPREDUCE-995:
-------------------------------------
I found this issue while preparing patch for Mumak (Hadoop Simulator MAPREDUCE-728). The backtrace of the problem is as follows:
{noformat}
java.lang.NullPointerException
at org.codehaus.jackson.impl.WriterBasedGenerator._verifyValueWrite(WriterBasedGenerator.java:553)
at org.codehaus.jackson.impl.JsonGeneratorBase.writeStartObject(JsonGeneratorBase.java:142)
at org.apache.hadoop.mapreduce.jobhistory.EventWriter.writeEventType(EventWriter.java:81)
at org.apache.hadoop.mapreduce.jobhistory.EventWriter.write(EventWriter.java:60)
at org.apache.hadoop.mapreduce.jobhistory.JobHistory.logEvent(JobHistory.java:339)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2519)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1130)
at org.apache.hadoop.mapred.SimulatorJobTracker.updateTaskStatuses(SimulatorJobTracker.java:621)
{noformat}
The last statement in the backtrace occurs in the context when the job has failed (one task fails 4 times), and the job is marked as complete. Then a lingering task that belongs to the same job completes, and JT tries to update its status.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756491#action_12756491 ]
Hong Tang commented on MAPREDUCE-995:
-------------------------------------
+1.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hong Tang updated MAPREDUCE-995:
--------------------------------
Priority: Critical (was: Major)
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756475#action_12756475 ]
Hong Tang commented on MAPREDUCE-995:
-------------------------------------
The bug seems trickier than I thought...
There is an obvious race condition:
Thread 1:
{code}
writer.close();
fileMap.get(id).clearEventWriter();
{code}
Thread 2:
{code}
MetaInfo mi = fileMap.get(jobId);
if (mi == null || (writer = mi.getEventWriter()) == null) {
{code}
writer.close() should be inside clearEventWriter(), and clearEventWriter() should close the writer and set it to null atomically. Another problem is that thread 2 may get a valid writer instance, but may later be closed by thread 1. So I think the right fix would have to synchronize on the MetaInfo for both clearing event writer and writing logs - logEvent would have to be implemented on MetaInfo, and synchornized with clearEventWriter().
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-995:
----------------------------------------
Attachment: mapred-995-v1.patch
Patch incorporating Hongs' comments.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jothi Padmanabhan updated MAPREDUCE-995:
----------------------------------------
Attachment: mapred-995.patch
Straight Forward Patch.
Also, made fileMap a ConcurrentHashMap.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Attachments: mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756930#action_12756930 ]
Chris Douglas commented on MAPREDUCE-995:
-----------------------------------------
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 2 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 release audit. The applied patch generated 175 release audit warnings (more than the trunk's current 0 warnings).
{noformat}
The release audit warning is clearly spurious.
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: M995-2.patch, mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated MAPREDUCE-995:
------------------------------------
Status: Patch Available (was: Open)
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: M995-2.patch, mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-995) JobHistory should handle cases
where task completion events are generated after job completion event
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756983#action_12756983 ]
Hudson commented on MAPREDUCE-995:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #48 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/48/])
. Fix a bug in JobHistory where tasks completing after the job
is closed cause a NPE. Contributed by Jothi Padmanabhan
> JobHistory should handle cases where task completion events are generated after job completion event
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-995
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-995
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jothi Padmanabhan
> Assignee: Jothi Padmanabhan
> Priority: Critical
> Attachments: M995-2.patch, mapred-995-v1.patch, mapred-995.patch
>
>
> It is apparently possible, in certain circumstances (failed job, for example), for the job history to get task completion events after the job completion event. This currently causes NPE in job history.
> Thanks Hong for identifying this issue
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.