You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2009/10/10 02:56:31 UTC

[jira] Created: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
-------------------------------------------------------------------------------

                 Key: MAPREDUCE-1089
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: contrib/fair-share
    Affects Versions: 0.21.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.

2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
java.lang.NullPointerException
        at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
        at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
        at java.util.Arrays.mergeSort(Arrays.java:1270)
        at java.util.Arrays.sort(Arrays.java:1210)
        at java.util.Collections.sort(Collections.java:159)
        at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
        at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
        at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771507#action_12771507 ] 

Hudson commented on MAPREDUCE-1089:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk #127 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/])
    . Fix NPE in fair scheduler preemption when tasks are  
scheduled but not running. Contributed by Todd Lipcon.


> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.21.0
>
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767547#action_12767547 ] 

Todd Lipcon commented on MAPREDUCE-1089:
----------------------------------------

Hi Matei,

I spent about 4 hours this morning attempting to get a unit test to demonstrate this behavior. Unfortunately it's very difficult since the current test doesn't separate the scheduling of a task and the beginning of the execution of that task. Despite my best efforts to get this working, I haven't been able to. At this point I can't dedicate any more time to it - how do you feel about committing as is without a test, or giving me some pointers on how to proceed with testing?

-Todd

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771233#action_12771233 ] 

Aaron Kimball commented on MAPREDUCE-1089:
------------------------------------------

Matei,

They changed the queues.xml format :P you need to run a 'git clean -f conf/' or an equivalent svn command to remove non-versioned files from your working directory. 'ant clean' doesn't cut it. Do that then run 'ant compile-core' again and you should be good to go again.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764377#action_12764377 ] 

Matei Zaharia commented on MAPREDUCE-1089:
------------------------------------------

The patch looks good, but I agree you should try to include a unit test.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767549#action_12767549 ] 

Todd Lipcon commented on MAPREDUCE-1089:
----------------------------------------

Manual test plan: set mapred.fairscheduler.update.interval and mapred.fairscheduler.preemption.interval both to very small numbers like 10ms, and then run some simple jobs. You will see NPEs in the logs with this patch not present.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matei Zaharia updated MAPREDUCE-1089:
-------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Reviewed]
           Status: Resolved  (was: Patch Available)

Thanks Aaron and Todd! I've committed this to trunk and to the 0.21 branch.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.21.0
>
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771284#action_12771284 ] 

Hudson commented on MAPREDUCE-1089:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #101 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/101/])
    . Fix NPE in fair scheduler preemption when tasks are  
scheduled but not running. Contributed by Todd Lipcon.


> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.21.0
>
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771225#action_12771225 ] 

Matei Zaharia commented on MAPREDUCE-1089:
------------------------------------------

Hey Todd,

I was going to commit your patch, but I'm getting a lot of fair scheduler test failures even without the patch due to "java.lang.RuntimeException: No queues defined" when initializing the JobTracker. I will wait until whoever broke this fixes it. Do you have any idea what patch caused this? I remember seeing a JIRA about this no queues issue before, but I can't find it.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated MAPREDUCE-1089:
-----------------------------------

    Attachment: mapreduce-1089.txt

Here's a patch to fix this issue. Unfortunately it's not triggered by the unit tests, since it uses a fake JobTracker which doesn't have quite the same behavior as the real one with regard to TaskStatus population. I'd like to get a unit test in that catches this issue before committing, but wanted to gather feedback on this patch now.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768130#action_12768130 ] 

Hadoop QA commented on MAPREDUCE-1089:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12421790/mapreduce-1089.txt
  against trunk revision 827884.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/196/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/196/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/196/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/196/console

This message is automatically generated.

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (MAPREDUCE-1089) Fair Scheduler preemption triggers NPE when tasks are scheduled but not running

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated MAPREDUCE-1089:
-----------------------------------

    Status: Patch Available  (was: Open)

> Fair Scheduler preemption triggers NPE when tasks are scheduled but not running
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1089
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1089
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/fair-share
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-1089.txt
>
>
> We see exceptions like this when preemption runs when a task has been scheduled on a TT but has not yet started running.
> 2009-10-09 14:30:53,989 INFO org.apache.hadoop.mapred.FairScheduler: Should preempt 2 MAP tasks for job_200910091420_0006: tasksDueToMinShare = 2, tasksDueToFairShare = 0
> 2009-10-09 14:30:54,036 ERROR org.apache.hadoop.mapred.FairScheduler: Exception in fair scheduler UpdateThread
> java.lang.NullPointerException
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1015)
>         at org.apache.hadoop.mapred.FairScheduler$2.compare(FairScheduler.java:1013)
>         at java.util.Arrays.mergeSort(Arrays.java:1270)
>         at java.util.Arrays.sort(Arrays.java:1210)
>         at java.util.Collections.sort(Collections.java:159)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasks(FairScheduler.java:1013)
>         at org.apache.hadoop.mapred.FairScheduler.preemptTasksIfNecessary(FairScheduler.java:911)
>         at org.apache.hadoop.mapred.FairScheduler$UpdateThread.run(FairScheduler.java:286)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.