You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Vinod K V (JIRA)" <ji...@apache.org> on 2009/03/02 11:58:12 UTC
[jira] Created: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
NPE in JobTracker.getTasksToSave() method
-----------------------------------------
Key: HADOOP-5374
URL: https://issues.apache.org/jira/browse/HADOOP-5374
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.20.0
Reporter: Vinod K V
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683784#action_12683784 ]
Amareshwari Sriramadasu commented on HADOOP-5374:
-------------------------------------------------
test-patch result:
{noformat}
[exec]
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
{noformat}
It is difficult to write testcase for this corner case.
All unit tests passed on my machine. Also Ran MRReliability with the patch.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682631#action_12682631 ]
Amareshwari Sriramadasu commented on HADOOP-5374:
-------------------------------------------------
Testfailure is not related to the patch. Raised HADOOP-5517 for the same.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5374:
--------------------------------------------
Status: Open (was: Patch Available)
cancelling patch, current code accepts FAILED_UNCLEAN/KILLED_UNCLEAN states and launches cleanup attempt, though previous state is FAILED. This should not happen, because FAILED->FAILED_UNCLEAN->FAILED updates will fail the task twice, thus doubling the counters and punishing TT twice.
I would say, It should not accept any state if the previous state is FAILED/KILLED.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5374:
--------------------------------------------
Status: Patch Available (was: Open)
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5374:
--------------------------------------------
Fix Version/s: 0.19.2
Affects Version/s: (was: 0.20.0)
0.19.0
Status: Patch Available (was: Open)
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677958#action_12677958 ]
Vinod K V commented on HADOOP-5374:
-----------------------------------
Here's the exact scenario:
- A TT gets a task but fails to report back immediately, making the JT mark the task as FAILED.
- The TT comes back with the task in COMMIT_PENDING state.
- JT doesn't update the task's state on itself as task cannot go from FAILED to COMMIT_PENDING state. (See state checks in TIP.updateStatus()). Thus TIP.doCommit() is not called.
- In getTasksToSave(), JT just checks the task's status on TT and tries to save the task, thus hitting NPE in TIP.shouldCommit().
Effect : The TT will keep getting NPE in heartbeat and become useless. The job will proceed as a duplicate task will be launched for this TIP.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-5374:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu reassigned HADOOP-5374:
-----------------------------------------------
Assignee: Amareshwari Sriramadasu
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5374:
--------------------------------------------
Attachment: patch-5374-1.txt
Patch with the fix.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-5374-1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677955#action_12677955 ]
Vinod K V commented on HADOOP-5374:
-----------------------------------
This affects heartbeats. I see a lot of the following in the JT's logs.
{code}
2009-02-28 08:12:10,332 INFO org.apache.hadoop.ipc.Server: IPC Server handler 34 on 52000, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@4b68032e, false, false, true, 5832) f
rom <ip:port>: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.mapred.TaskInProgress.shouldCommit(TaskInProgress.java:420)
at org.apache.hadoop.mapred.JobTracker.getTasksToSave(JobTracker.java:2585)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2334)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
{code}
An offline discussion with Amareshwari concluded that in getTasksToSave() method, the task's state on the JT should also be checked along with the state as reported by the TT.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-5374:
--------------------------------------------
Attachment: patch-5374-2.txt
Attaching the patch with fix.
Tested the patch with ExpireLaunchingTasks thread marking the task FAILED, and TT reporting COMMIT_PENDING, FAILED_UNCLEAN, KILLED_UNCLEAN, FAILED, KILLED and SUCCEEDED states for the task.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683815#action_12683815 ]
Vinod K V commented on HADOOP-5374:
-----------------------------------
Looked at the patch. Looks good to me. +1
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682571#action_12682571 ]
Amareshwari Sriramadasu commented on HADOOP-5374:
-------------------------------------------------
There are three threads in JobTracker updating the TaskStatuses : ExpireTrackers, ExpireLaunchingTasks and heartbeat threads.
1. If heartbeat runs first, there is no issue with ExpireTrackers and ExpireLaunchingTasks threads, since the entry will removed from them.
2. If ExpireTrackers runs first, there is no issue with heartbeat(The tracker will be re-inited) and ExpireLaunchingTasks(he task-entry will removed from the thread).
3. If ExpireLaunchingTasks runs first,
a) If ExpireTrackers runs second, the task attempt will be Killed again. The task will have FAILED, followed by KILLED update.
b) If heartbeat runs second, If the TaskStatus is UNASSIGNED, RUNNING, COMMIT_PENDING or SUCCEDED, the task will be added to taskToKill map and the update is ignored. So, the tasks will be sent KillTaskAction(fixed by HADOOP-5280). If the state is COMMIT_PENDING, it throws NPE as described in this jira.
I propose this issue can fix both 3(a) and 3(b).
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Attachments: patch-5374-1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12685409#action_12685409 ]
Hadoop QA commented on HADOOP-5374:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402624/patch-5374-2.txt
against trunk revision 756752.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/115/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/115/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/115/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/115/console
This message is automatically generated.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12689223#action_12689223 ]
Hudson commented on HADOOP-5374:
--------------------------------
Integrated in Hadoop-trunk #790 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/790/])
. Fixes a NPE problem in getTasksToSave method. Contributed by Amareshwari Sriramadasu.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt, patch-5374-2.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5374) NPE in JobTracker.getTasksToSave()
method
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682620#action_12682620 ]
Hadoop QA commented on HADOOP-5374:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402354/patch-5374-1.txt
against trunk revision 755057.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/94/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/94/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/94/console
This message is automatically generated.
> NPE in JobTracker.getTasksToSave() method
> -----------------------------------------
>
> Key: HADOOP-5374
> URL: https://issues.apache.org/jira/browse/HADOOP-5374
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.2
>
> Attachments: patch-5374-1.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.